AVCaptureDeviceDiscoverySession & stream continuity across physical hinge transitions with Virtual Front Camera

Question

JulienKermarec OP

Created 9h

Replies 2

Boosts 0

Participants 2

This post is from the Activity iPhone Duo Photos & Camera Q&A.

Hello team,

When configuring an AVCaptureMultiCamSession or consuming the new Virtual Front Camera, what is the recommended architecture to avoid frame drops or pipeline rebuilds (AVCaptureSessionRuntimeError) during dynamic posture transitions (from folded to unfolded / half-open)?

Specifically:

Does the Virtual Front Camera handle the lens handover internally with continuous timestamps in CMSampleBuffer, or does the underlying device switch trigger a device disconnect/reconnect notification that requires re-attaching inputs?

How are the optical intrinsics (matrix_float3x3) and lens distortion coefficients updated through the hinge motion? Can we rely on continuous metadata attachments on each frame during the transition, particularly if running real-time CoreML / Vision processing?

Thanks!

Answer 1

Frameworks Engineer OP

Apple

8h

When using the Virtual Front Camera, you'll see frame drops when the app switches between the physical outer and inner front cameras.

Optical intrinsics and lens distortion coefficients should get updated as the Virtual Front Camera switches cameras.

Answer 2

JulienKermarec OP

8h

Thank you for clarifying!

Following up on the frame drop during the switch:

Does the session drop frames silently at the AVCaptureVideoDataOutput level, or will it explicitly fire a CMSampleBuffer with the .droppedFrameReason attachment (or post an AVCaptureSessionWasInterruptedNotification)?

What is the typical handover latency (in frames or ms) developers should account for to freeze/gracefully cross-fade the camera preview during posture transitions?