I got it working after copying all sample buffer metadata. Somehow it must have been modified somewhere in my code and that occasionally made recording break.

Just tested switching mic using the Control Center in iOS 26, and that works! I noticed the new AVInputPickerInteraction in iOS 26, which looks promising, but there is a slight catch. My app is a streaming app, and users would like to switch mic automatically when switching "scene/camera". It's not suitable to present a picker every time that happens, so would be awesome if the mic selection could be done in code (in Swift).