Post

Replies

Boosts

Views

Activity

Reply to iOS: Recording from two AVCaptureSessions is out of sync
Oh my god, I finally found the issue. The Video Stabilization mode cinematicExtended actually caused a huge delay in the Video Pipeline because it keeps an internal buffer of frames to stabilize them. This just takes time. Apparently this takes up to 1 second on my modern iPhone 15 Pro. The way my RecordingSession was designed is that I just wrote the Video and Audio buffers to the session once I received them, and I used their presentation timestamps as the timestamp for the video. What I didn't account for here was that the presentation timestamp is not the same value as the current time, the frame might be older and we just got it after it has went through a bunch of processing (stabilization)! E.g. relative to the timer we see on screen - when I start recording, the time is 5:00, but the first frame that actually arrives at 5:00 still has a presentation timestamp of 3:86, because that's when it was captured from the Camera. The mistake I previously made here was that we wrote this Frame to the file - which I shouldn't do, it's from the past! This PR fixes that behaviour and only writes Frames to the file that have a presentation timestamp later than when we actually started recording, so the first frame that should've been written to the file was the one with the presentation timestamp 5:00, which might come in when the current time is already 6:86, so it's 1:86 seconds too late. Here's my PR with code which fixes the issue: https://github.com/mrousavy/react-native-vision-camera/pull/2206 What a funny bug, I was pulling my hairs out on this one.
Topic: App & System Services SubTopic: Core OS Tags:
Nov ’23
Reply to Performance issues with `AVAssetWriter`
Some additional information: Regarding the audio AVAssetWriter.init(...) call taking 3 seconds: These are the recommendedVideoSettingsForAssetWriter settings I get: ["AVVideoCompressionPropertiesKey": { AllowFrameReordering = 1; AllowOpenGOP = 1; AverageBitRate = 47848572; BaseLayerFrameRate = 15; ExpectedFrameRate = 60; MaxAllowedFrameQP = 41; MaxKeyFrameIntervalDuration = 1; MinAllowedFrameQP = 15; MinimizeMemoryUsage = 1; Priority = 80; ProfileLevel = "HEVC_Main_AutoLevel"; RealTime = 1; RelaxAverageBitRateTarget = 1; }, "AVVideoWidthKey": 2160, "AVVideoCodecKey": hvc1, "AVVideoHeightKey": 3840] (video initializes in ~10ms) And these are the recommendedAudioSettingsForAssetWriter(..) I get: ["AVEncoderQualityForVBRKey": 91, "AVEncoderBitRatePerChannelKey": 96000, "AVEncoderBitRateStrategyKey": AVAudioBitRateStrategy_Variable, "AVFormatIDKey": 1633772320, "AVNumberOfChannelsKey": 1, "AVSampleRateKey": 44100] (audio initializes in ~1500-3000ms) Note that subsequent calls are faster. I am also changing the AVAudioSessionCategory while configuring the audio AVAssetWriter, not sure if that has anything to do with it. Regarding the AVAssetWriter.startWriting() call taking 3-5 seconds: I still have no idea why that takes so long.
Nov ’23
Reply to Combining Matched Geometry with New Zoom Navigation Transitions in iOS 18
Hey - did you find a solution to this? I'm wondering the same - View A on source screen always zooms to become the destination screen, instead of zooming into View B on the destination screen.
Topic: UI Frameworks SubTopic: SwiftUI Tags:
Replies
Boosts
Views
Activity
Mar ’25
Reply to Interpreting received "Data" object in cpp
Out of curiosity; can Data be somehow bridged to C++ so I can hold on to it longer and access bytes and size at a later point?
Topic: Programming Languages SubTopic: Swift Tags:
Replies
Boosts
Views
Activity
Dec ’24
Reply to Camera Video Orientation
Meanwhile the actual pixel data in both cases is oriented as landscapeLeft How did you figure this one out? Does the CMSampleBuffer have a "orientation" property or something?
Topic: Media Technologies SubTopic: Audio Tags:
Replies
Boosts
Views
Activity
Jun ’24
Reply to iOS: Recording from two AVCaptureSessions is out of sync
Oh my god, I finally found the issue. The Video Stabilization mode cinematicExtended actually caused a huge delay in the Video Pipeline because it keeps an internal buffer of frames to stabilize them. This just takes time. Apparently this takes up to 1 second on my modern iPhone 15 Pro. The way my RecordingSession was designed is that I just wrote the Video and Audio buffers to the session once I received them, and I used their presentation timestamps as the timestamp for the video. What I didn't account for here was that the presentation timestamp is not the same value as the current time, the frame might be older and we just got it after it has went through a bunch of processing (stabilization)! E.g. relative to the timer we see on screen - when I start recording, the time is 5:00, but the first frame that actually arrives at 5:00 still has a presentation timestamp of 3:86, because that's when it was captured from the Camera. The mistake I previously made here was that we wrote this Frame to the file - which I shouldn't do, it's from the past! This PR fixes that behaviour and only writes Frames to the file that have a presentation timestamp later than when we actually started recording, so the first frame that should've been written to the file was the one with the presentation timestamp 5:00, which might come in when the current time is already 6:86, so it's 1:86 seconds too late. Here's my PR with code which fixes the issue: https://github.com/mrousavy/react-native-vision-camera/pull/2206 What a funny bug, I was pulling my hairs out on this one.
Topic: App & System Services SubTopic: Core OS Tags:
Replies
Boosts
Views
Activity
Nov ’23
Reply to Frames out of order using AVAssetWriter
Hey - did you find a solution for this? In my case, video and audio tracks are out of sync by almost a full second...
Topic: Graphics & Games SubTopic: Metal Tags:
Replies
Boosts
Views
Activity
Nov ’23
Reply to Performance issues with `AVAssetWriter`
Some additional information: Regarding the audio AVAssetWriter.init(...) call taking 3 seconds: These are the recommendedVideoSettingsForAssetWriter settings I get: ["AVVideoCompressionPropertiesKey": { AllowFrameReordering = 1; AllowOpenGOP = 1; AverageBitRate = 47848572; BaseLayerFrameRate = 15; ExpectedFrameRate = 60; MaxAllowedFrameQP = 41; MaxKeyFrameIntervalDuration = 1; MinAllowedFrameQP = 15; MinimizeMemoryUsage = 1; Priority = 80; ProfileLevel = "HEVC_Main_AutoLevel"; RealTime = 1; RelaxAverageBitRateTarget = 1; }, "AVVideoWidthKey": 2160, "AVVideoCodecKey": hvc1, "AVVideoHeightKey": 3840] (video initializes in ~10ms) And these are the recommendedAudioSettingsForAssetWriter(..) I get: ["AVEncoderQualityForVBRKey": 91, "AVEncoderBitRatePerChannelKey": 96000, "AVEncoderBitRateStrategyKey": AVAudioBitRateStrategy_Variable, "AVFormatIDKey": 1633772320, "AVNumberOfChannelsKey": 1, "AVSampleRateKey": 44100] (audio initializes in ~1500-3000ms) Note that subsequent calls are faster. I am also changing the AVAudioSessionCategory while configuring the audio AVAssetWriter, not sure if that has anything to do with it. Regarding the AVAssetWriter.startWriting() call taking 3-5 seconds: I still have no idea why that takes so long.
Replies
Boosts
Views
Activity
Nov ’23
Reply to AVAssetWriter leading to huge kernel_task memory usage
Hey - a bit unrelated to the memory leak, but in my case calling AVAssetWriter.startWriting() takes around 3 seconds to complete on a modern iPhone 14 - is there any way to speed that up? See my post: https://developer.apple.com/forums/thread/741942 Thanks!
Topic: Media Technologies SubTopic: Audio Tags:
Replies
Boosts
Views
Activity
Nov ’23