Thank you for the context @galad87!
I did suspect issues with the presentation timestamp, but unfortunately have been unable to recreate the issue by artificially messing with the timestamps. The relevant method is
append(_ pixelBuffer: CVPixelBuffer, withPresentationTime presentationTime: CMTime) on AVAssetWriterInputPixelBufferAdaptor.
From my experimentation, artificially delaying a timestamp just causes a delay in the footage (somewhat obvious), while setting timestamps in the past just causes the writer to fail since AVAssetWriter expects frames in order. I have not been able to find a combination of timestamps to force the reordering behavior described before. Something could definitely still be wrong with our implementation, but I am starting to wonder if something is going on with AVAssetWriter internally.