Building Dasher (eye-gaze text-entry system for people with motor impairments — ALS, cerebral palsy, etc.) for visionOS. See https://github.com/dasher-project/Dasher-Apple The app needs continuous (x, y) gaze coordinates to steer a zooming alphabet interface. Vision Pro has incredible eye tracking hardware built in — this should be the perfect platform — but I can't get gaze coordinates through any API I've tried. What I've tested (all deliver zero events in a windowed app): UIHoverGestureRecognizer — Apple's docs say it "detects when the user is looking at a view" on visionOS. Added it to a UIView inside UIViewRepresentable. Looked at the view for 2+ minutes on real hardware. Console.app shows zero began/changed/ended callbacks. .onContinuousHover (SwiftUI) — zero .active phases delivered, whether attached to the canvas view or the ZStack root. UIPointerInteraction with a delegate — regionFor callback never fires. The system clearly is tracking gaze — I can see beginScrollingWithRegion events in Console.app with the correct window rect, and the visual hover highlight appears on buttons. But the coordinates are routed to scroll handling, not exposed as a usable pointer position. What does work: pinch-and-drag follows the hand position (where the pinch is in 3D space), not gaze. That's fine for general apps but doesn't help users who need eye-only control. What I'm wondering: Has anyone successfully received continuous gaze coordinates in a windowed (non-immersive) visionOS app? If so, which API? Is UIHoverGestureRecognizer supposed to work on UIView inside UIViewRepresentable, or does it require a pure UIKit view hierarchy? Is there an ARKit provider for eye tracking that I'm missing? I see HandTrackingProvider but no EyeTrackingProvider in the capability list. Could the issue be that my SwiftUI overlay (toolbar VStack) is intercepting hover events before they reach the canvas underneath? If so, how do you make a SwiftUI overlay pass-through for hover? Dasher is a well-established accessibility tool that lets people type using only their eyes. Its the fastest text entry method for eyegaze and head controlled systems. On every other platform (iOS with external trackers, Windows with Tobii, etc.) it reads continuous gaze coordinates and zooms toward wherever the user is looking. Vision Pro should be the best hardware for this — the tracking is already there, we just can't read it. If there's no public API for this, I'd love to know so I can scrap my feedback assistant entry!. If there IS a way and I've missed it, I'd be grateful for a pointer. Thanks!

How do I snap a window to a surface in the visionOS Simulator?

Reality Composer Pro 3 Assistant appears to have useful project-aware tool categories for working with scenes, assets, materials, animation, behavior trees, navigation, skeletons, audio, planning, documentation, and ShaderGraph material creation. It would be very helpful if a supported subset of these capabilities could be exposed to external developer tools through an Apple-provided MCP server, command-line tool, or similar local automation interface. Similar to what's possible with Xcode: https://developer.apple.com/documentation/Xcode/giving-external-agents-access-to-xcode Possible exposed capabilities could include: Querying and modifying scene hierarchy, entities, and components. Inspecting and managing project assets and structure. Working with materials and ShaderGraph/SSL material creation. Reading and editing animation libraries, timelines, skeletons, and rigs. Inspecting behavior trees and navigation data. Reading audio components and playback settings. Querying Reality Composer Pro documentation through the Assistant. Returning structured JSON responses for external tools. Letting external coding agents request project-aware plans or safe editor actions. Expected Result: Reality Composer Pro provides a documented, permission-aware MCP server, CLI, or similar interface that allows external developer tools to safely use supported Assistant/project capabilities. Actual Result: Reality Composer Pro Assistant is available only inside the app UI. External coding agents such as Codex or Claude Code cannot reliably inspect or automate Reality Composer Pro projects except through unsupported package/file manipulation. Why This Matters: Many developers now use coding agents alongside Xcode and Reality Composer Pro. A supported interface would make app development smoother by allowing those tools to coordinate Swift code, RealityKit assets, scene state, materials, animations, and Assistant workflows through an official integration point. Feedback ticket: FB23276072

Reality Composer Pro Assistant currently appears limited to a fixed OpenAI model list and the legacy /v1/chat/completions endpoint. On Reality Composer Pro 3.0 (75.0.14.500.1), the OpenAI provider exposes only a small set of models such as gpt-4.1, gpt-4o, gpt-4o-mini, o1, and o3-mini. Newer models available through OpenAI’s current APIs are not usable. It would be nice to add support for current provider APIs, including OpenAI’s /v1/responses endpoint, and allow dynamic model discovery or manual custom model IDs. Feedback ticket: FB23274502

I had planned on using the new version of Reality Composer Pro to export USDZ files for interactivity, but I noticed it can only export selected objects, and they export to a new '.reality' file type. I can't open these on MacOS26.6 or MacOS27. On the old version of Reality Composer Pro you could export the entire project as a USZ file, which was really cool since it was kind of universal and maintained many of the interactions. is this something that just hasn't been integrated yet?

Before WWDC I had built a fairly simple tap-to-open suitcase where you could tap on an object inside to activate it. (Play audio, no are, etc) all of this was managed in the timeline, which no longer exists. This was really nice because the usdz file was easily shared as a storytelling tool. in the new version… do I really need a model with a closed state, an open state and an opening and closing animations to do something like this? Is there a transform I can perform on an object with a tap instead of creating a specific animation separately? And then using script graphing and animation graphing? I appreciate the in-depth tutorials to showcase the awesomefeatures, but I’m finding it hard to take the information for something as basic like this out of them. even a simple description of how to tap and open a door would make things easier to understand.

I created a minimal sample project that reproduces a microphone issue with nearby SharePlay / Window Sharing on Apple Vision Pro. Sample project: https://github.com/JerryNee/mic-demo Screen recording: https://youtu.be/aMj_A_leJWU Issue I first implemented SharePlay through FaceTime and found that local microphone capture was unavailable. That seemed understandable because FaceTime itself uses the microphone. I then moved the app to a nearby SharePlay / Window Sharing session specifically to avoid a FaceTime call, but local microphone capture still fails. Local microphone capture works before starting nearby SharePlay. After starting a nearby SharePlay / Window Sharing session with GroupActivities, the same app fails at: try audioEngine.start() This does not require a FaceTime call. It happens with an in-room nearby SharePlay / Window Sharing session. The screen recording also goes silent when the Share Window UI appears. I checked the MP4 audio track with ffmpeg silencedetect; audio becomes silent at about 00:00:22.60, matching the moment nearby sharing starts: silence_start: 22.600612 silence_end: 55.051927 | silence_duration: 32.451315 Environment Two nearby Apple Vision Pro devices Xcode device list: visionOS 27.0 and visionOS 26.5 Xcode 27.0, build 27A5194q Frameworks: GroupActivities, AVFoundation, Speech Entitlement: com.apple.developer.group-session Microphone and speech recognition permissions granted Steps Run the sample on Apple Vision Pro. Tap Start Listening and speak. Transcription works. Tap Stop Listening. Tap Start Nearby Session. Share with another nearby Apple Vision Pro. Tap Start Listening again while the nearby session is active. audioEngine.start() fails. Ending the nearby session makes microphone capture work again. Minimal microphone code let recognizer = SFSpeechRecognizer() let audioEngine = AVAudioEngine() let request = SFSpeechAudioBufferRecognitionRequest() request.shouldReportPartialResults = true let session = AVAudioSession.sharedInstance() try session.setCategory(.record, mode: .measurement, options: [.duckOthers]) try session.setActive(true) let input = audioEngine.inputNode let format = input.outputFormat(forBus: 0) input.installTap(onBus: 0, bufferSize: 1024, format: format) { buffer, _ in request.append(buffer) } audioEngine.prepare() try audioEngine.start() Minimal nearby SharePlay code struct MicDemoActivity: Codable, GroupActivity, Transferable { static let activityIdentifier = "com.example.mic-demo.nearby-mic-repro" static var transferRepresentation: some TransferRepresentation { CodableRepresentation(contentType: .micDemoActivity) } var metadata: GroupActivityMetadata { var metadata = GroupActivityMetadata() metadata.type = .generic metadata.title = "Mic Demo Nearby Session" return metadata } } for await session in MicDemoActivity.sessions() { session.join() } _ = try await MicDemoActivity().activate() Questions Is local microphone capture expected to be unavailable during a nearby SharePlay / Window Sharing session? Is screen recording audio expected to become silent during nearby sharing? What is the recommended way to support nearby shared UI/state while still allowing local voice input?

How to make RCP start playing Timeline from scratch when receiving OnNotification. At present, I have only played the timeline once after receiving the OnNotification, and cannot replay it again after receiving the OnNotification again.

I have been working with visionOS 27's website environment features, and I have found when entering a website environment, leaving the website environment, waiting 10 seconds, and then trying to re-enter the website environment, the attempt fails. The thrown error reports: AbortError: Immersive request was superseded by another request Strangely, if I try to immediately re-enter the website environment, I succeed. I have to wait several seconds before trying to re-enter the environment to experience the failure. The workaround is to reload the page. Anyone know why this is occurring or how to avoid it? Anyone know a more graceful workaround than asking the user to reload the page? I have filed a feedback on this: FB23208345

Hi there! While building my visionOS app, I’ve encountered some strange behaviour with EnvironmentBlendingComponent. Entities using .occluded(by: .surroundings) are culled entirely whenever a Mac Virtual Display or Apple Immersive Environment is behind them, regardless of the entities’ actual depth, and even when the environment is set to coexist. This feels like it could be a bug, although I’m fairly new to visionOS development, so I may be missing something! In a .mixed immersive space with plain passthrough, the component behaves as I would expect: an entity is occluded accurately when a real-world object is placed in front of it. However, when either of the following is visible behind the entity: a Mac Virtual Display window; or an Apple Immersive Environment enabled using the Digital Crown, the entity carrying EnvironmentBlendingComponent disappears completely. This happens even when the entity is physically closer to the viewer than the virtual surface. An otherwise identical entity without the component remains visible. Moving the virtual surface out of the line of sight, disabling the Immersive Environment, or removing the component immediately makes the entity visible again. Is it intended that .occluded(by: .surroundings) treats Mac Virtual Display and Apple Immersive Environments as occluders? The documentation describes this mode as depth-based occlusion against static “real-world objects”, so it isn’t clear whether these system-rendered virtual surfaces should participate. Even if they are intended to participate, the observed behaviour appears depth-independent: the entire entity is hidden when the virtual surface is behind it, rather than only when the surface is closer to the viewer. That seems inconsistent with the documented depth-based behaviour. I can reproduce this on an M5 Apple Vision Pro running visionOS 26.5 using the following minimal example: import SwiftUI import RealityKit import UIKit @main struct OcclusionReproApp: App { var body: some SwiftUI.Scene { WindowGroup { OcclusionReproLauncher() } ImmersiveSpace(id: "immersive") { RealityView { content in // Control: opaque, no component — proves the bare setup renders. let control = ModelEntity( mesh: .generateSphere(radius: 0.12), materials: [SimpleMaterial(color: .green, isMetallic: false)] ) control.position = [-0.25, 1.2, -1.3] // ~1.3 m ahead, eye height // Under test: identical opaque sphere plus the occlusion component. let occluded = ModelEntity( mesh: .generateSphere(radius: 0.12), materials: [SimpleMaterial(color: .red, isMetallic: false)] ) occluded.position = [0.25, 1.2, -1.3] occluded.components.set( EnvironmentBlendingComponent(preferredBlendingMode: .occluded(by: .surroundings)) ) content.add(control) content.add(occluded) } } .immersionStyle(selection: .constant(.mixed), in: .mixed) // Let the app's mixed space coexist with an Apple Immersive Environment, // as visionOS suppresses Environments while an immersive space is open otherwise. // The same bug can be reproduced with just a Mac Virtual Display, but // this provides a second way to reproduce the same behaviour: .immersiveEnvironmentBehavior(.coexist) } } struct OcclusionReproLauncher: View { @Environment(\.openImmersiveSpace) private var openImmersiveSpace var body: some View { Button("Open immersive space") { Task { await openImmersiveSpace(id: "immersive") } } .padding() } } To reproduce (on a Vision Pro, as the Simulator won't provide passthrough): Open the immersive space in plain passthrough. Both spheres should be visible. Place a real object in front of the red sphere. It should be occluded correctly according to depth. Open a Mac Virtual Display or enable an Apple Immersive Environment, with its visible surface behind the red sphere. The red sphere disappears entirely, despite being in front of that surface. The green control sphere remains visible. Move the virtual surface out of view or disable it, and the red sphere reappears. Please let me know if this is expected behaviour, or if I'm doing something wrong - thanks! Jack

For the Reality Composer Pro Assistant that generates 3D objects and materials: what's the commercial-use and licensing status of generated assets in shipped apps, and what data underpins the model? Alternatively, am I able to upload an original image and use that for 3D generation?

How should I set the window of WindowGrop to resemble a curved screen style?

As the title says. I am developing an app for IOS 18, which involves adding a building relatively far away from the user's place (around 10-20m.), and I need to building to show contact shadows with the floor for added realism. Now, I know that RealityKit has automatic contact shadows using GroundingShadowComponent, but this only works if the object is placed in a plane that has been detected by ARKit... and ARKIt doesn't detect floors so far away, so I need to add my own shadows: add an invisible plane acting as a floor below my building, and have it receive shadows and show only the shadows. The problem is that, from what I see, RealityKit has no materials that can do this: UnlitMaterial ignores all lighting, including shadows. SimpleMaterial does display the shadows, but the floor is displayed too. I can make the material almost transparent (setting the opacity to 0.01 or something), but the floor is still visible. OcclusionMaterial with receiveDynamicLighting is a solution, but it doesn't work either. If I declare it using OcclusionMaterial(receivesDynamicLighting: true), all I get is an invisible plane with no contact shadows in it. What do other people do for this? Do people just bake in the contact shadows in the 3D model?

I got it working recently that I could be in an immersive environment in my App with another FaceTime participant while we both see each others personas. The room needed to be shared as it has a presentation baked in to it that one person can present. However, since yesterday whenever I enter the immersive SharePlay experience, the personas no longer appear and only a coin of the other person is visible with the text "immersed" beneath it. The room is still shared, so I can change the slides and the other person can see the changes. If one person exits the immersive space, their persona is rendered from my point of view (in the immersive space) but we can't be in there together.

Summary We have identified a reproducible world tracking drift regression in ARKit on LiDAR-equipped iOS devices running iOS 26.4 and later. A static virtual node anchored at the world origin visually drifts from its initial position as the user moves around a real-world scene, despite the scene remaining physically static. The same code produces stable, drift-free results on non-LiDAR devices running identical OS versions. Device & OS Observations Testing was performed across four devices on iOS 26.4 using the same application build and ARWorldTrackingConfiguration settings. Non-LiDAR devices — iPhone 14 and iPhone 15 — produced stable, drift-free tracking in all test runs. No world origin displacement was observed regardless of how long or how far the user walked. LiDAR-equipped devices — iPhone 14 Pro and iPhone 16 Pro — exhibited consistent, reproducible drift. A static node placed at the world origin visually shifted from its initial position as the user moved through the scene. The same devices were stable on earlier iOS versions, confirming this is a regression introduced in iOS 26.4. Technical Observations Nature of drift: A SCNNode placed statically at the ARKit world origin (SCNVector3(0, 0, 0)) visually displaces from its original position as the user walks around a static real-world scene. The displacement is not random — it accumulates directionally as the user moves, consistent with a sensor fusion or coordinate anchoring error. Trigger condition: The drift occurs during normal walking motion around a fixed point of interest, such as circling a parked vehicle. It does not appear when the device is held still. LiDAR specificity: The drift is exclusive to devices with a LiDAR scanner. Identical hardware configurations — same iOS build, same ARWorldTrackingConfiguration settings — on non-LiDAR devices produce no drift whatsoever. This isolates the regression to the LiDAR sensor's contribution to ARKit's internal Visual-Inertial Odometry (VIO) fusion pipeline. No API-level workaround found: There is currently no public ARKit API to selectively disable the LiDAR scanner's contribution to VIO. All available configuration-level options have been evaluated without resolving the drift. ARWorldTrackingConfiguration Options Evaluated The following configuration changes were applied individually and in combination. None resolved the drift on LiDAR devices: isAutoFocusEnabled = false — No improvement videoHDRAllowed = false (disabled) — No improvement planeDetection = [] (disabled) — No improvement sceneReconstruction = .mesh — No improvement worldAlignment: .gravity vs .gravityAndHeading — No improvement Minimal Reproduction Case The drift can be reproduced with a minimal ARKit scene: Create an ARSCNView with ARWorldTrackingConfiguration using default settings. Add a single static SCNNode (e.g., a small sphere or axes geometry) at SCNVector3(0, 0, 0) when the session starts. Run the app on a LiDAR-equipped device (iPhone Pro, iPad Pro with LiDAR) on iOS 26.4 or later. Walk in a circle around the node's approximate real-world position. Expected: The node remains visually fixed at its world position throughout the walkthrough. Actual: The node drifts from its initial position, increasingly displaced from its world origin anchor as walking continues. We want to know if Apple has made any internal updates to ARKit, particularly after the OS 26.4 upgrade. Thanks!

We currently use MapKit as our map layer for our application. We have 3D models of buildings and other features that we want to overlay and visualize on the map. The issue seems to be that - as far as I can tell - we can only pitch the map to whatever iOS decides is the "right" pitch for the map, not what our imagery has. So if we wanted to pitch the map even when 3D imagery isn't available it doesn't seem like that is possible. Are there options for us to provide our own 3D models / imagery in MapKit? Or is there something else we should be looking at?

I've been trying to implement the new Spatial Preview framework in my app but I've been unable to make it work. I then tried the official sample: https://developer.apple.com/documentation/spatialpreview/working-with-content-from-your-mac-app-using-spatial-preview But this sample is also not able to stream anything to my Vision Pro. Both my Mac & Vision Pro are on the same network with bluetooth enabled, I'm using Mac Virtual Display but the "Stream USD Scene to a vision pro" does nothing. Mac is on macOS 27 and Vision Pro on visionOS 27, I'm using XCode 27. So everything should be setup correctly here. Log says: {CSInlineDonation[async]: "com.example.apple-samplecode.SpatialPreviewSampleB9WT723MN2" add-update-items:1 delete-items:0}: Failed to request donation Error Domain=CSIndexErrorDomain Code=-1000 "Failed to request donation" UserInfo={NSDebugDescription=Failed to request donation, NSUnderlyingError=0x778d201b30 {Error Domain=NSCocoaErrorDomain Code=4099 "The connection to service named com.apple.SetStoreUpdateService was invalidated from this process." UserInfo={NSDebugDescription=The connection to service named com.apple.SetStoreUpdateService was invalidated from this process.}}} Just submitted a ticket about this issue: FB23028075 Am I doing something wrong here? Anyone got the sample working?

It would be nice to be able to round the corners of images that are displayed using the ImagePresentationComponent in VisionOS 26. Quick Look and all the native photo apps have that aesthetic, so the sharp square corners can look out of place.

I have been building a visionOS app that converts anaglyphs and SBS to 3D experiences, and my current approach has been hamstrung by my dependency on a shader graph which separates imagery into right and left eye. It works very well, but ultimately is just a material. The way I have implemented this so far is with VNC to show a 3d Blender window and browser experiences. Ideally, I'd be able to have interactivity in both that didn't depend on an attached computer. Is there any way to accomplish that?

This is my animation playback method, there are two issues. 1.How to make the animation stop until the last frame after playing once when isLoop=false? When I specify the name of an animClipName animation clip, I cannot play the corresponding animation name. How do I set it up? The animation structure is shown in the figure. PlayAnim(animEntityName: "Book",animClipName: "Open",isLoop: false) private func PlayAnim(animEntityName: String,animClipName: String? = nil,isLoop:Bool = true,transitionDuration: Double = 0.5){ guard let XR = XR else { return } guard let entity = XR.findEntity(named: animEntityName) else { return } let availableAnims = entity.availableAnimations let targetAnimResource: AnimationResource? if let clipName = animClipName { targetAnimResource = availableAnims.first(where: { $0.name == clipName }) } else { targetAnimResource = availableAnims.first } guard let animClip = targetAnimResource else { return } let anim = animClip.repeat(count: isLoop ? 0 : 1) entity.playAnimation(anim,transitionDuration: transitionDuration) }