We are developing a LiDAR-based spatial capture application using ARKit and RealityKit.
We are investigating whether PhotogrammetrySession can be used as part of a capture pipeline where we collect:
- RGB frames
- LiDAR depth
- camera intrinsics
- camera transforms
- IMU/device motion data
and then reconstruct a textured 3D model.
The documentation states that PhotogrammetrySample can include depthDataMap, depthConfidenceMap, gravity, and camera metadata.
Could someone clarify how PhotogrammetrySession uses the depth information when supplied?
Specifically:
- Is LiDAR depth used during reconstruction to improve geometry (for example as a constraint during reconstruction), or is it primarily used for scale/orientation metadata?
- When camera transforms and intrinsics are available from ARKit, does PhotogrammetrySession consume this information, or does it independently estimate camera poses from the image sequence?
- Is the recommended Apple workflow for RGB + LiDAR capture:
ARKit capture → PhotogrammetrySample → PhotogrammetrySession
- or is ObjectCaptureSession the intended capture pipeline for photogrammetry workflows?
We are trying to understand the intended relationship between ARKit, ObjectCaptureSession, and PhotogrammetrySession before choosing an architecture.
Thanks.
Thanks for asking. This sounds like an interesting project. Here is a brief survey of what I was able to find ~
One distinction in the documentation is worth separating first, because the rest follows from it. Depth appears in two places, described as doing two different things.
For a PhotogrammetrySample you construct yourself, the documented role of depthDataMap is scale. The property documentation says that providing depth data "can help PhotogrammetrySession determine the real-world scale of the photographed object and result in a correctly sized 3D object for placement in an AR scene." The WWDC24 session Discover area mode for Object Capture lists the sample inputs the same way, describing "a depth map for recovering metric scale, a gravity vector for uprighting, a metadata dictionary of extra information, and of course an optional object mask."
The LiDAR data that ObjectCaptureSession collects is described separately, and it is not necessarily the same input. The WWDC23 session Meet Object Capture for iOS describes reconstructing a low-texture chair: "In addition to RGB images, our API also collects point cloud data with LiDAR, which helps to produce a comprehensive representation of the object's 3D shape with enhanced coverage and density. Finally, a complete 3D model is generated from the fused-point cloud data."
Those are two different data paths. depthDataMap is a property you set on a sample you build. The LiDAR point cloud is gathered by ObjectCaptureSession and stored in the images it writes. A point cloud appears in the PhotogrammetrySession API as an output, through a pointCloud request. There is no public API for supplying one as an input.
On camera transforms and intrinsics, the API surface gives a direct answer. A session takes its input in one of two forms, a URL or a sequence of PhotogrammetrySample values. PhotogrammetrySample has four settable properties: metadata, depthDataMap, gravity, and objectMask. Its id and image are set at initialization. The remaining properties, including camera and depthConfidenceMap, are read-only, and PhotogrammetrySample.Camera has no public initializer. So there is no public API for setting camera transforms or intrinsics on a sample you construct.
The WWDC24 session describes what those read-only properties are for: "we provide several new read-only properties that let you load data saved by our ObjectCaptureSession capture UI. These might be useful in an inspector UI or for your own custom reconstruction pipelines." It adds that images captured with ObjectCaptureSession will have the camera transform available for each sample. The intrinsics matrix and any available calibration data are provided as well. That data is there for your code to read.
Camera pose is also available as an output. A poses request returns PhotogrammetrySession.Poses, documented as a "mapping from the sample ID to the 6DOF algorithmically estimated pose of that sample."
On how the pieces relate, the ObjectCaptureSession documentation states the division: "Once a session enters the .completed state, your app can transfer the images to a Mac or use them locally on the iOS device for use in object reconstruction using PhotogrammetrySession. Model reconstruction is a separate phase which this session does not directly monitor or control."
ARKit is referenced on the capture side only. ObjectCaptureSession.cameraTracking is documented as "the current state of ARKit camera tracking", and its Tracking documentation describes the ARKit coaching overlay appearing automatically when tracking degrades. Neither the PhotogrammetrySession nor the PhotogrammetrySample documentation mentions ARKit.
One further constraint may affect the architecture choice. On-device reconstruction offers a single detail level. The WWDC23 session states "we support only the reduced detail level on iOS". In the iOS SDK, PhotogrammetrySession.Request.Detail declares reduced, and the preview, medium, full, raw, and custom cases are each marked unavailable on iOS. Those higher detail levels are available when reconstructing on a Mac.
