In Apple's Camera app (set to "Photo" mode), when the system switches between different cameras, the transition is incredibly smooth. In particular, when getting too close to an object for the wide angle camera (due to the minimal focus distance), it automatically switches to the ultra-wide angle. When switching, the app transitions between the cameras very smoothly, by which I mean the position of the object in frame doesn't shift despite the physical location of the cameras being different. Is it possible to replicate this behaviour with the AVFoundation API?
When creating a AVCaptureSession using a AVCaptureDevice with the builtInTripleCamera device type, the session automatically switches between the constituent cameras as expected. However, unlike Apple's Camera app, there is a very noticeable shift in perspective when switching. This is especially noticeable when switching between the WA and UWA cameras while zoomed in, sometimes resulting in the center of the frame being shifted all the way to the edge of the screen.
I was able to somewhat replicate the smooth switching by setting activeVideoStabilizationMode of my AVCaptureConnection (with AVCaptureVideoDataOutput) to .cinematic, but this would introduce too much delay for a live preview, and none of the other available stabilization modes seemed to support this smooth transitioning.
Is it possible to replicate the behaviour of the Camera app using the AVFoundation API?
Video stabilization modes like .cinematic use device calibration to align the cameras during switchover and the added latency allows it to smooth the output trajectory to minimize shifts. You could implement a low latency version using the same device calibration, which is available by querying +[AVCaptureDevice extrinsicMatrixFromDevice:toDevice:].
For better alignment between cameras you can try streaming two at a time using AVCaptureMultiCamSession and periodically apply image based registration.
If you’d like the built-in transitions to be smoother please file an enhancement request.
