Well, according "Body Tracking Example Using RealityKit, ARKit 4 and SwiftUI"  video on youtube by Ryan Kopinsky, the coordination system for body is same as world coordination system except origin changed. So the position calculation is simply vector addition.

Just want to add to the description: the direction of arms and hands looks accurate. But the skeleton model of arm/body length was off.

 aharriscrowne and TikiMcFee , thank you for your reply. They are helpful. My work was interrupted recently. But I am picking it up again and I will update here.

It appears that if I use .right orientation and I am using iPad in portrait way, the image and the detected points aligns without conversion (points detected with vision are the points used for raycast. let handler = VNImageRequestHandler(cvPixelBuffer: frame.capturedImage, orientation: .right, options: [:])