Solved it. As detailed in this WWDC talk, - https://developer.apple.com/videos/play/wwdc2019/225/ you have to correct for the intrinsic matrices to relate video frames from the respective cameras to each other. Oddly, it seems that AVCaptureConnection's isCameraIntrinsicMatrixDeliverySupported returns false unless both connections have been added to the session, in which case it returns true (given the session topology in question). The intrinsic matrices of the respective cameras do change in real time, which accounts for the unexpected vertical disparity. This WWDC talk describes - https://developer.apple.com/videos/play/wwdc2017/507/ the intrinsic matrix in a bit more detail.

In case anyone comes across this thread and was having the same problem I was: - https://stackoverflow.com/a/66585389/222262 When capturing from both constituent devices in builtInDualCamera, I had to enable intrinsics delivery on each connection after both connections were added to the session. Before that, isCameraIntrinsicMatrixDeliverySupported would return false for both connections. Not sure how/if this insight applies to a single-cam case.

From 36:28, this WWDC talk strongly - https://developer.apple.com/videos/play/wwdc2019/225/ implies you need to correct for the constituent devices' intrinsic matrices in order to relate the images to each other. This is probably the issue. Unfortunately, isCameraIntrinsicMatrixDeliverySupported returns false for both of my connections, even though my capture session is set up exactly as described in the talk. Does anyone know why isCameraIntrinsicMatrixDeliverySupported would be false? To see my capture session topology, please check out the slides on AVDualCam shown in the WWDC talk above.