Currently VisionOS has WorldAnchors which are a great, privacy-preserving method to affix and localize entities into physical spaces. These are great for apps that work in Mixed immersive mode as well as augmented layer or virtual overlays onto physical spaces. However, they have some fundamental limitations when compared to iOS’s ARWorldMap which limit their capabilities as truly persistent WorldAnchors, which the platform would greatly benefit from. The problem - Impermanence: WorldAnchors are on-device only, stored in data at a system level and expose their ID for collection and usa within apps. If the app which created them is removed, or if the device is rese, those saved World Anchors are removed and lost forever. Their ID will no longer point to an anchor in physical space and it is as if the anchor has been deleted or never existed. This means that anchors effectively only temporarily exist on a single device while that specific device is in an undisturbed state. A second device using that same app with interest in utilizing that anchor is not available. With no way to export this Anchor, save it, re-load it either inside an app bundle, from a server, on-site data storage, etc. the anchor is 'trapped' on the device which creates it. ARWorldMap can be saved to a file, and acquired in many ways which circumvents this problem, and allows it to exist as a single source of truth for that particular physical environment that any iOS or iPadOS device can localize to as long as they have the ARWorldMap. Imperfect Workaround #1 - SharedAnchors: After setting a WorldAnchor on Device #1, start a local S harePlay session with Device #1 & #2 in the same physical space. Then, use a 'Shared Anchor' with transform offset data matching the WorldAnchor. On Device #2, save that offset as a new WorldAnchor. This new WorldAnchor will have the same transform effectively matching the original, though it will have a new ID and is entirely separate. Problem with Workaround #1: This requires two VisionOS devices to be in the same physical location simultaneously, have an active SharePlay session between the two devices, then conduct a sharing operation. This means that if Device #1 does not happen to be available in the physical space at the same time as Device #2 for any reason, Device #2 will never have access to this anchor. In cases such as a public or industrial space, it is not realistically viable to always have a person with Device #1 available at all times. There are many situations where Device #2 would want to enter the space and observe the anchor while they are alone, and would never have access to this anchor. Workaround #2 - iOS / iPadOS Middleman: Once a visionOS WorldAnchor is created on Device #1, that same user can manually co-locate an iOS or iPadOS device, with some shared session (for example, using ImageTrigger on the iOS/iPadOS device's display which the visionOS device reads, then determines an offset for). Then, create an ARWorldMap on that co-located iOS/iPadOS device, and save it, then serve that ARWorldMap. Then to localize visionOS Device #2, enter the space and first localize an iOS / iPadOS device with the ARWorldMap. Then, manually co-locate visionOS Device #2 with the iOS/iPadOS device and access the offset. Then save this offset as WorldAnchor on Device #2, at this point the iOS/iPadOS device is no longer needed. Problems With Workaround #2: At a minimum, this requires 3 devices, or more realistically 4 if Device #1 and #2 are not in the space at the same time. This is not only a very poor user experience to have to operate an iOS/iPadOS device simultaneously while wearing visionOS device, but it is also very complicated and a several step process that is not intuitive to most users. The accuracy of this process is extremely poor, as tracking an image from the screen of an iOS/iPadOS device will never be as precise as the internal tracking system on VisionOS. It will always have SOME margin for drift and error, which can result in the anchor being very far offset from the intended anchor position. This is antithetical to the purpose of WorldAnchors, as the delta can be so inconsistent that Device #2 will observe content attached to the anchor at an incorrect location, which could lead to unintended user behaviour when interacting with this content potentially moving attached content to look correct for them, which would negatively impact all other devices viewing the content. Desired Solution: An optimal upgrade to visionOS WorldAnchors would; Use the same underlying tracking & relocalization system that is currently in use Allow for exporting of the WorldAnchor's data in an encrypted, privacy preserving way (not simply a point cloud) which could be saved and shared as a file similar to ARWorldMap, then used for relocalization on ANY device with access to that file and application. Be interoperable across platforms where iOS, iPadOS, visionOS and any other spatial-capable platform can use that single file, and localize themselves in the physical space. This would unify ARKit WorldAnchors for all platforms, ensuring that the same physical space can be localized to by all devices, anchors can be created by all devices, and the content would exist in the same position on all devices. Allowing an iOS or iPadOS device to create a WorldAnchor that is then identical on visionOS, or visa-versa. These features unlock true persistence in anchoring content to real-world spaces, a critical component of a Spatial Computing platform that maximizes the unique capabilities and benefits of the medium. This creates an upwards path for users to start on iOS / iPadOS today, and upgrade to visionOS in the future. Accessibility: Not all users are able to use visionOS for various reasons, including physical, regional and financial circumstances among many many others. There is no reason why someone who is unable to use visionOS for any one of these reasons should be LOCKED-OUT from Spatial Computing applications. Spatial computing is the most human computing medium ever created, and applications need to allow all humans to engage with Spatial Computing experiences regardless of their level of access to visionOS devices. We as developers want to build everlasting Spatial Computing applications that accentuate the medium, maximize the benefits of Spatial Computing, include / invite all humans regardless of their Accessibility level, and establish virtual content that can outlive the individuals who create it. Please, take this request into serious consideration, as the feature as described would contribute to the Apple Ecosystem being the single greatest Spatial Computing platform of all time (and space), enabling permanent layering of physical spaces, preserving privacy for sensitive data, and maximizing accessibility across the spectrum of humans and devices. Thank you.

This is a very general and high level question. I am working on an app that scans buildings using on device LiDAR. One of my biggest issues is drift over time. What is the recommended approach by Apple to successfully scan for a long period of time and reduce drift error? I am currently storing frames at 20 fps with pose data and then trying to correct for pose errors off device. Maybe my approach is incorrect and would like to understand how Apple recommends to do about this. I am being overly general on purpose. Thank you for your help.

We are building an ARKit application targeting iPhone Pro devices on iOS 26.4+. During testing, we consistently observe the following log message at a specific point in our AR session (approximately 90° into a walking arc around a vehicle): ARWorldTrackingTechnique: resource constraints [33] What specifically does resource constraint code [33] represent? Is it a VIO constraint type, a sensor fusion budget limit, or something else entirely? How does it differ from code [1] (which appears to be a general CPU starvation signal for the ARKit sensor fusion thread)? Is there a public API to detect or respond to this specific constraint condition, short of waiting for a full ARCamera.TrackingState downgrade? Was this constraint type introduced or changed in iOS 26.4?

I'm trying to track identified objects in realtime with bounding rects, with no 3D integration, but still has poor update performance. I'm trying to understand how to optimize frame updates. (I'm a new programmer) Using: Foundation, AVFoundation, ARKit, CoreVideo, Vision, CoreImage, CoreVideo with YOLOE-11s object detection currently throttled to 2fps. (target iOS, testing on 16pro) YOLOE-11S CoreML model detects objects with class labels + bounding boxes Labels are matched against ObjectCatalog.json for relevance Matched objects are promoted from blue (detected) to green (identified) Log warnings: ARSession <0x110afdb80>: The delegate of ARSession is retaining 13 ARFrames. The camera will stop delivering camera images if the delegate keeps holding on to too many ARFrames. This could be a threading or memory management issue in the delegate and should be fixed. Skipping integration due to poor slam at time: 619447.208339 vio_initialized(1) map_size(0) tracking_state_is_nominal(0) is_3dof(0) reinitialize_attempts(6) slam_mode(RegularSLAM)

Film crews record "takes" in my app: camera feed plus virtual AR content with timecode. On SceneKit I rely on a third-party Metal-layer hook (SCNRecorder), because ReplayKit captures overlay UI and adds a permission prompt — and RealityKit appears to have no capture API at all, making recording a regression in any SceneKit→RealityKit migration. Is there a recommended first-party way to record an AR view to a movie file, or should I file feedback?

My users import their own 3D assets at runtime — OBJ, FBX, DAE, STL, PLY, often with skeletal animation — which we convert in-app to SceneKit scenes via Assimp. RealityKit only loads USD/.reality at runtime. Is the recommended path to hand-build MeshResource, skeletons, and animations from parsed geometry, or is there any supported on-device conversion to USD? This is one of my biggest blockers to leaving SceneKit.

My app is an 8-year-old film-previsualization tool built on ARSCNView, with SceneKit rendering over an ARKit world-tracking session. Given SceneKit's deprecation with critical-bug-only maintenance: does that commitment, and the "ample notice" promise, explicitly cover ARSCNView? Should I plan on existing SceneKit apps continuing to work on new hardware and OS releases for several more years? I have a few blockers stopping me from migrating to RealityKit and I've noticed the lack of big updates to ARKit over the recent years.

On iOS I colocate multiple iPhones/iPads by sharing ARWorldMap over MultipeerConnectivity. visionOS 26 added the conceptual equivalent (shared coordinate spaces via SharedCoordinateSpaceProvider) but as far as I can tell it's Vision Pro-to-Vision Pro only, and visionOS has no ARWorldMap, so there's no bridge. Is there any supported way today to put an iPhone and a Vision Pro into one shared coordinate space or should I file feedback for cross-platform colocation? It would be useful to my users, for example, to have someone controlling an AR scene layout on a Vision Pro while using iOS devices to preview and record the scene from various specific angles.

Is simultaneous ProRes Log 422 capture and ARKit VIO tracking supported on iPhone? If not today, is this a capability Apple is considering for future SDK releases? I’m interested in capturing high-quality ProRes Log footage and, at the same time, using ARKit for real-time camera pose tracking for synchronization and post-processing workflows.

My app previews cinema lenses by cropping the AR feed, so I can only simulate lenses tighter than the main camera's ~24mm-equivalent, meaning wider cinema lenses are impossible to accurately preview. ARConfiguration.VideoFormat exposes captureDeviceType, but world-tracking formats only ever list the wide-angle module. Is running world tracking on (or alongside) the ultra-wide camera fundamentally infeasible? Or just a case of not enough Feedback Assistant requests?

Hello, I’m developing an Apple Vision Pro app designed to help people with mobility and speech challenges communicate more effectively with their caregivers. The app requires camera access in order to function as intended. I previously submitted a request for the required camera access entitlement by email and received an initial response indicating that the request would be reviewed with a colleague, but I have not received any follow-up since then. My Apple Developer account is registered under an LLC, and I currently have other visionOS apps available on the App Store. Additional documentation for this app is available here: https://revivrstudios.com/stareandshare.html My company is focused on developing Apple Vision Pro apps that improve quality of life for people living with mobility and speech challenges. I believe this app has the potential to provide meaningful support for those users and their caregivers. Could someone advise on the best way to follow up on the camera access request, or confirm whether there is an additional process I should complete? Thank you.

Hi, We're building a mixed-immersive visionOS app in which the user holds a thin metal tool — a needle — near virtual content. We need the real needle to correctly occlude virtual content: when the needle (and especially its tip) is physically in front of a virtual object, the virtual object must not draw over it. We are solution-agnostic. Whether the right answer is LiDAR / scene-depth based, object-tracking based, hand-tracking based, or something else entirely — we just need needle occlusion to work. Here is what we've ruled out so far: ObjectTrackingProvider — blocked at enrolment. We can't even produce a usable reference object: the object-capture / scanning step doesn't pick up the needle at all (≈1 mm shaft, specular/metallic, almost no surface features). So object tracking never gets off the ground — it fails before runtime. SceneReconstructionProvider + OcclusionMaterial — only usable for large planar surfaces. Occlusion against a wall or floor is clean. But the mesh is too coarse for anything with finer geometry: a chair already gives very rough, ragged occlusion edges, and the needle is far below the mesh resolution — its shaft never appears in the mesh, so it never occludes at all. So this isn't only a needle problem: it's a mesh resolution problem that goes from rough (chair) to nonexistent (needle). Automatic hand occlusion — doesn't cover the tip. The hand is composited in front correctly, but the needle tip protrudes a few centimetres beyond the fingers, and virtual content draws over exactly that part. The needle tip is the part that most needs to occlude correctly, so this is blocking for us. The full Xcode sample project and the screen recording are in this GitHub repo: https://github.com/JerryNee/occlusion-demo (the video is at media/occlusion-demo.mp4). The screenshots below are stills from that recording, which has four stages, all with the same virtual cube: Occlusion off — baseline; the cube draws over everything in the room. Occlusion on, against a wall — the cube is occluded cleanly. Occlusion on, against a chair — the cube is occluded, but with rough, ragged edges. Occlusion on, with a needle — the needle does not occlude the cube at all. The same occlusion path therefore degrades from clean (wall) → rough (chair) → nonexistent (needle) as the real geometry gets finer. Questions / avenues we'd welcome guidance on (any one would unblock us): What is the recommended way to occlude an object this thin on visionOS — and is it possible at all today? Is there app-accessible per-pixel scene depth (analogous to ARFrame.sceneDepth on iOS) that we could use to occlude geometry the reconstruction mesh misses? Can the system's automatic hand / upper-limb occlusion be extended to include a thin instrument held in the hand — i.e. treat a held tool as part of the hand region? The tool has known, rigid geometry, so we considered attaching an OcclusionMaterial proxy mesh and positioning it from hand-tracking joints — but the needle's orientation within the grip is not determined by the hand pose (the same grip can hold the needle at many different angles), so a hand-joint proxy would be misaligned. Is there any supported way to recover the actual 6-DoF pose of a thin tool held in the hand (i.e. something that senses the tool itself, since the hand pose doesn't constrain it)? Is there any way to make object capture / scanning succeed for thin, specular objects, or an alternative enrolment path? Environment: Apple Vision Pro, visionOS 26.2. Full Xcode sample + screen recording: https://github.com/JerryNee/occlusion-demo Even a "not currently possible / known limitation" answer would help us plan our roadmap. Thank you!

Hi all: Novice iOS programmer here. I am developing an iPhone app (target: iOS 18) where I need to place a large building a bit far away from the user: around 10-20 meters away. Now, the problem I am having is that ARKit's floor detection usually doesn't reach that far. I solved it by using "existingPlaneInfinite" in the raycast: let raycastResults = arView.session.raycast( ARRaycastQuery(origin: rayOrigin, direction: rayDirection, allowing: .existingPlaneInfinite, alignment: .horizontal) But the problem is that the building appears... but without grounding shadows. I tried enabling GroundingShadowComponent, but without success. After some more testing, I found out that if I place an object in a plane that ARKit is actually detecting, then it will have grounding shadows... but if I place it further away, in the part of the floor that ARKit doesn't detect, then it won't add shadows. It will still place the building at the right height (because it extends the detected plane to the infinite), but it won't add contact shadows with the floor. I can solve this problem using the usual tricks (adding an invisible plane as a shadow catcher, etc.), but I just wanted some confirmation that this is indeed the correct behaviour. Also, are there any changes in this regard in IOS 26?

With the recent announcement that VisionOS 27 would support custom actively tracked tools, via arrays of embedded LEDs, I would just like to confirm if the demo given in this video here: https://developer.apple.com/videos/play/wwdc2026/283/ at minute 1:00 is indeed an actively tracked tool, or if it is passive IR. if it's passive IR (reflectance), are there any examples of how this could be achieved, i.e. can we access the onboard IR cameras directly?

Hello, we are evaluating Apple Vision Pro for an AR calibration experiment. Can BarcodeDetectionProvider on visionOS 2.0+ detect a QR code/barcode and provide its 3D position and orientation, such as a transform, so that we can align virtual content with a real-world marker? Does the BarcodeAnchor update continuously when the headset or the barcode moves? Thank you.

Hello, I am doing a personal research project around ARKit camera frames and camera calibration data. For each accepted ARFrame, I export: ARFrame.timestamp ARFrame.camera.imageResolution ARFrame.camera.intrinsics ARFrame.capturedImage The target video mode is 4K / 30 FPS. In my test captures, I noticed two things: ARFrame.camera.intrinsics is not constant across the recording. Some neighboring frames from ARFrame.capturedImage have noticeably different sharpness: one frame can look sharp, while the next frame can look blurred. Regarding intrinsics, I observe small per-frame changes in fx/fy/cx/cy. For example, in one session: fx/fy changed by approximately 36 px cx changed by approximately 1.5 px cy changed by approximately 2.3 px I understand the standard meaning of the intrinsic matrix: fx / fy = focal length in pixels cx / cy = principal point in pixels relative to the image/reference frame My main question is not about the definition of these values, but about how to correctly interpret their changes over time. In particular, I am interested in cx/cy, because the principal point is important for measurements. I would like to understand whether a change in cx/cy corresponds to a meaningful update of the camera model for the current frame, and what physical or camera-pipeline process may cause this change. Questions about ARFrame.camera.intrinsics: Are ARFrame.camera.intrinsics values expected to change between frames during a single ARWorldTrackingConfiguration session? If fx/fy changes over time, can this reflect autofocus / focus breathing, internal camera calibration updates, digital crop/scaling, stabilization-related mapping, or other camera pipeline changes? If cx/cy changes over time, should I interpret this as an updated principal point for the current ARFrame image/reference frame? Does ARKit update cx/cy to account for any internal crop, scaling, stabilization, virtual camera behavior, lens movement, or calibration changes? Is there a known physical interpretation for small cx/cy changes reported by ARFrame.camera.intrinsics, or should these values simply be treated as ARKit’s best current camera model for that frame? In an ARKit-first pipeline where focus/exposure/stabilization are managed by ARKit/camera pipeline, is the recommended approach to store ARFrame.camera.intrinsics per frame together with ARFrame.timestamp and ARFrame.camera.imageResolution? Is there any public API that exposes the reason for intrinsics changes, such as focus position, stabilization transform, crop transform, active camera constituent, lens movement, or per-frame calibration update reason? I also have a related question about ARFrame.capturedImage frame quality. In some recordings, neighboring frames can have noticeably different sharpness. For example, one frame can look sharp, while the next frame can be visibly blurred, even though the scene and camera movement are continuous. Questions about frame sharpness / camera pipeline behavior: Is frame-to-frame sharpness variation expected when using ARFrame.capturedImage as the video source? Can ARKit change focus, exposure duration, ISO, white balance, crop/scaling, or other camera pipeline parameters between frames during an ARWorldTrackingConfiguration session? Is there any public API to inspect per-frame exposure duration, ISO, focus position, lens position, stabilization state, crop/scaling transform, or other capture parameters for ARFrame.capturedImage? If some frames are sharp and neighboring frames are blurred, should this usually be interpreted as motion blur / exposure behavior / autofocus behavior, or can ARKit internal processing also affect this? Is there any recommended way in an ARKit-first pipeline to reduce frame-to-frame sharpness inconsistency, or is the only reliable approach to use an AVFoundation-first capture pipeline with locked focus/exposure/ISO/white balance? The main goal of my research is to understand how to correctly interpret ARKit-provided intrinsics over time and how much of the camera pipeline behavior is observable through public ARKit APIs. Since fx/fy/cx/cy are important for measurements, I want to know whether treating ARFrame.camera.intrinsics as the Apple-delivered per-frame camera model is the safest approach. Thank you.

Hi everyone, I’m currently using the RoomPlan API, which has been working reliably until recently. However, I’ve started encountering an intermittent error and I’m trying to understand what might be causing it. The error is triggered in the ARSession observer method: session(_ session: ARSession, didFailWithError error: Error) It has occurred on at least two devices: iPhone 14 Pro iPhone 17 Pro Here’s the full error message: ARSession failed domain=com.apple.arkit.error code=102 desc=Required sensor failed. userInfo=["NSLocalizedFailureReason": A sensor failed to deliver the required input., "NSUnderlyingError": Error Domain=AVFoundationErrorDomain Code=-11819 "Cannot Complete Action" UserInfo={NSLocalizedDescription=Cannot Complete Action, NSLocalizedRecoverySuggestion=Try again later.}, "NSLocalizedDescription": Required sensor failed.] This seems to indicate that a required sensor (likely LiDAR or camera) failed to provide input, but I’m not sure what’s causing it or why it happens only occasionally. Has anyone experienced something similar or has insight into possible causes or fixes? Thanks in advance!

ARCamera.TrackingState questions: Did the threshold or sensitivity for transitioning ARCamera.TrackingState from .normal to .limited(.excessiveMotion) or .limited(.insufficientFeatures) change between iOS 26.3 and iOS 26.4? What does "ARWorldTrackingTechnique: resource constraints [33]" mean, and is it new in iOS 26.4? Does it correspond to a tracking state degradation? Is there a way for the client to detect or respond to ARKit entering a resource-constrained mode short of the full tracking state transition — for example, a lower-level notification or a flag on ARFrame — so that apps can take protective action without interpreting it as a full tracking failure? ARCamera.unprojectPoint questions: Did the behavior of ARCamera.unprojectPoint(_:ontoPlane:orientation:viewportSize:) change between iOS 26.3 and iOS 26.4 for near-parallel geometry? Specifically, on iOS 26.3 this method returns nil when the camera ray is nearly parallel to the target plane (denominator of the ray-plane intersection → 0 at ~90° of camera rotation). On iOS 26.4, with identical code and environment, it returns a large finite value instead — we observed z = −12.27m. Since the method's optional return type implies nil is the documented signal for no valid intersection, this reads as a behavioral regression rather than an intentional change. If returning the computed value for near-parallel geometry is now the intended behavior, is there a recommended way for the caller to guard against it? For example, should we check abs(dot(rayDirection, planeNormal)) against a threshold before calling, and if so, is there a documented epsilon Apple uses internally? Alternatively, is there a newer API we should prefer over unprojectPoint(:ontoPlane:) for this use case that handles degenerate geometry more gracefully — such as ARSession.raycast(:)? Are there any other ARKit API adjustments between OS 26.3 and 26.4? We are using the same codebase, but it behaves differently in between these 2 OS versions now. Thanks!

I have two RealityView: ParentView and When click the button in ParentView, ChildView will be shown as full screen cover, but the camera feed in ChildView will not be shown, only black screen. If I show ChildView directly, it works with camera feed. Please help me on this issue? Thanks. import RealityKit import SwiftUI struct ParentView: View{ @State private var showIt = false var body: some View{ ZStack{ RealityView{content in content.camera = .virtual let box = ModelEntity(mesh: MeshResource.generateSphere(radius: 0.2),materials: [createSimpleMaterial(color: .red)]) content.add(box) } Button("Click here"){ showIt = true } } .fullScreenCover(isPresented: $showIt){ ChildView() .overlay( Button("Close"){ showIt = false }.padding(20), alignment: .bottomLeading ) } .ignoresSafeArea(.all) } } import ARKit import RealityKit import SwiftUI struct ChildView: View{ var body: some View{ RealityView{content in content.camera = .spatialTracking } } }

I have an iOS app that uses RealityView to display some models and interact with them, and the app uses regular iOS app navigations, then a challenge I'm facing is how to maintain multiple RealityView on multiplescreens. For example Screen A has a RealityView, and then I navigate to Screen B (also has a RealityView) using stack based navigation, when I do so I got a crash -[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5970: failed assertion `Draw Errors Validation Fragment Function(fsRealityPbr): argument envProbeTable[0] from Buffer(7) with offset(0) and length(16) has space for 16 bytes, but argument has a length(864). Fragment Function(fsRealityPbr): incorrect type of texture (MTLTextureType2D) bound at Texture binding at index 20 (expect MTLTextureTypeCubeArray) for envProbeDiffuseArray[0]. Interestingly this crash only happens when debugging with Xcode, not happens when the app runs on its own. I'm not sure what I'm doing is anti-pattern or it's some Xcode debugging limitation.