With the recent announcement that VisionOS 27 would support custom actively tracked tools, via arrays of embedded LEDs, I would just like to confirm if the demo given in this video here: https://developer.apple.com/videos/play/wwdc2026/283/ at minute 1:00 is indeed an actively tracked tool, or if it is passive IR. if it's passive IR (reflectance), are there any examples of how this could be achieved, i.e. can we access the onboard IR cameras directly?
Passive IR Tracking on VisionOS 27
Hi @cdog123
The demo uses object tracking, not passive IR. With object tracking, you train a reference object in Create ML from a USDZ of the physical object, and ARKit's ObjectTrackingProvider detects and tracks it in the wearer's surroundings. No markers, reflective or otherwise, are involved.
As of visionOS 27, Create ML produces reference objects capable of more accurate and lower-latency tracking. To pick up the improvements, retrain your reference objects with the latest version of Create ML. For handheld or moving objects, also look at:
- The Extended training mode in Create ML, intended for the highest tracking quality on handheld objects.
- highFrameRateTrackingEnabled on ReferenceObject.Configuration, which opts in to high frame-rate tracking at the cost of additional power.
There is no public API to access the IR cameras or other sensors on the device for custom tracking. If neither object tracking nor the spatial accessory APIs cover your use case, I'd encourage you to file a request through Feedback Assistant describing what you're trying to build, why the existing APIs don't fit, and what a successful outcome would look like.
Further reading
For object tracking (what's used in the demo), including the visionOS 27 updates:
For accessory tracking (active):