Background: I am prototyping with RealityKit with ios 14.1 on a latest iPad Pro 11 inches. My goal was to track a hand. When using skeleton tracking, it appears skeleton scales were not adjusted correctly so I got like 15cm off in some of my samples. So I am experimenting to use Vision to identity hand and then project back into 3D space. 1> Run image recognition on ARFrame.capturedImage let handler = VNImageRequestHandler(cvPixelBuffer: frame.capturedImage, orientation: .up, options: [:]) let handPoseRequest = VNDetectHumanHandPoseRequest() .... try handler.perform([handPoseRequest]) 2> Convert point to 3D world transform (where the problem is).    fileprivate func convertVNPointTo3D(_ point: VNRecognizedPoint,                     _ session: ARSession,                     _ frame: ARFrame,                     _ viewSize: CGSize)               -> Transform?   {     let pointX = (point.x / Double(frame.camera.imageResolution.width))*Double(viewSize.width)     let pointY = (point.y / Double(frame.camera.imageResolution.height))*Double(viewSize.height)     let query = frame.raycastQuery(from: CGPoint(x: pointX, y: pointY), allowing: .estimatedPlane, alignment: .any)     let results = session.raycast(query)     if let first = results.first {       return Transform(matrix: first.worldTransform)     }     else {       return nil     }   } I wonder if I am doing the right conversion. The issue is, in the ARSession.rayCast document - https://developer.apple.com/documentation/arkit/arsession/3132065-raycast, it says this is converting UI screen point to 3D point. However, I am not sure how ARFrame.capturedImage will be fit into UI screen. Thanks

I wrote/copied a code to show major body joint locations. The goal was to track hand location. The hand locations seem off quite a bit. Please let me know what I could change to make it better. My initial impression was that the skeleton seemed not adjusted to a person's body type - in my case, I was shorter than standard skeleton model. Device: iPad Pro (11-inch) (2nd generation) Software Version: 14.0 (18A373) XCode Version: 12.0(12A7209) iOS Deployment Target: 14.0 I was standing there without obstruction and I was not moving for a couple of seconds when taking screenshot. I could not attach screenshot. But by my visual estimation, the hand joint locations were about 10-15cm away. Here is how I coded it - 1> create an entity for each joint interested 2> they are all added to a "VisualSkeleton" (extension of Entity) object 3> Create an AnchorEntity and Place this Entity to the anchorEntity; 4> Refresh each ModelEntity's location based on corresponding joint's location Configurating ...         // Run a body tracking configration.         let configuration = ARBodyTrackingConfiguration()         configuration.automaticImageScaleEstimationEnabled = true         configuration.automaticSkeletonScaleEstimationEnabled = true         arView.session.run(configuration) Calculates joint positions     func update(with bodyAnchor: ARBodyAnchor) {         let rootPosition = simd_make_float3(bodyAnchor.transform.columns.3)         let skeleton = bodyAnchor.skeleton                  //rootAnchor.position = rootPosition         //rootAnchor.orientation = Transform(matrix: bodyAnchor.transform).rotation                  for (jointName, jointEntity) in joints {             if let jointTransform = skeleton.modelTransform(for: ARSkeleton.JointName.init(rawValue: jointName)) {                 let jointOffset = simd_make_float3(jointTransform.columns.3)                 jointEntity.position = rootPosition + jointOffset // rootPosition                 jointEntity.orientation = Transform(matrix: jointTransform).rotation             }         }                  if self.parent == nil {             rootAnchor.addChild(self)         }     } I will be happy to share more code if needed. Thank you so much!