ScreenCapture + CMSampleBuffer logic issue

i'm trying to work on a simple screen recording app on macOS that always records the last 'x' seconds of your screen and saves it whenever you want, as a way to get comfortable with swift programming and apple APIs.

i was able to get it running for the past '30 seconds' and record and store it. however i realised that there was a core issue with my solution: i was defining the SCStreamConfiguration.queueDepth = 900 (to account for 30fps for 30 seconds) which goes completely against apple's instructions: https://developer.apple.com/documentation/screencapturekit/scstreamconfiguration/queuedepth?language=objc

now when i changed queueDepth back to 8, i am only able to record 8 frames and it saves only those first 8 frames. i am unsure what the flow of the apis should be while dealing with screenCaptureKit.

for context, here's my recording manager code that handles this logic (queueDepth = 900)

import Foundation
import ScreenCaptureKit
import AVFoundation

class RecordingManager: NSObject, ObservableObject, SCStreamDelegate {
    static let shared = RecordingManager()
    
    @Published var isRecording = false
    private var isStreamActive = false // Custom state flag
    
    private var stream: SCStream?
    private var streamOutputQueue = DispatchQueue(label: "com.clipback.StreamOutput", qos: .userInteractive)
    private var screenStreamOutput: ScreenStreamOutput? // Strong reference to output
    private var lastDisplayID: CGDirectDisplayID?
    private let displayCheckQueue = DispatchQueue(label: "com.clipback.DisplayCheck", qos: .background)
    
    // In-memory rolling buffer for last 30 seconds
    private var rollingFrameBuffer: [(CMSampleBuffer, CMTime)] = []
    private let rollingFrameBufferQueue = DispatchQueue(label: "com.clipback.RollingBuffer", qos: .userInteractive)
    private let rollingBufferDuration: TimeInterval = 30.0 // seconds
    
    // Track frame statistics
    private var frameCount: Int = 0
    private var lastReportTime: Date = Date()
    
    // Monitor for display availability
    private var displayCheckTimer: Timer?
    private var isWaitingForDisplay = false

    func startRecording() {
        print("[DEBUG] startRecording called.")
        guard !isRecording && !isWaitingForDisplay else {
            print("[DEBUG] Already recording or waiting, ignoring startRecording call")
            return
        }
        isWaitingForDisplay = true
        isStreamActive = true // Set active state
        checkForDisplay()
    }
    
    private func setupAndStartRecording(for display: SCDisplay, excluding appToExclude: SCRunningApplication?) {
        print("[DEBUG] setupAndStartRecording called for display: \(display.displayID)")
        let excludedApps = [appToExclude].compactMap { $0 }
        let filter = SCContentFilter(display: display, excludingApplications: excludedApps, exceptingWindows: [])
        let config = SCStreamConfiguration()
        config.width = display.width
        config.height = display.height
        config.minimumFrameInterval = CMTime(value: 1, timescale: 30) // 30 FPS
        config.queueDepth = 900
        config.showsCursor = true
        print("[DEBUG] SCStreamConfiguration created: width=\(config.width), height=\(config.height), FPS=\(config.minimumFrameInterval.timescale)")
        stream = SCStream(filter: filter, configuration: config, delegate: self)
        print("[DEBUG] SCStream initialized.")
        self.screenStreamOutput = ScreenStreamOutput { [weak self] sampleBuffer, outputType in
            guard let self = self else { return }
            guard outputType == .screen else { return }
            guard sampleBuffer.isValid else { return }
            guard let attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, createIfNecessary: false) as? [[SCStreamFrameInfo: Any]],
                  let statusRawValue = attachments.first?[.status] as? Int,
                  let status = SCFrameStatus(rawValue: statusRawValue),
                  status == .complete else {
                return
            }
            self.trackFrameRate()
            self.handleFrame(sampleBuffer)
        }
        do {
            try stream?.addStreamOutput(screenStreamOutput!, type: .screen, sampleHandlerQueue: streamOutputQueue)
            stream?.startCapture { [weak self] error in
                print("[DEBUG] SCStream.startCapture completion handler.")
                guard error == nil else {
                    print("[DEBUG] Failed to start capture: \(error!.localizedDescription)")
                    self?.handleStreamError(error!)
                    return
                }
                DispatchQueue.main.async {
                    self?.isRecording = true
                    self?.isStreamActive = true // Update state on successful start
                    print("[DEBUG] Recording started. isRecording = true.")
                }
            }
        } catch {
            print("[DEBUG] Error adding stream output: \(error.localizedDescription)")
            handleStreamError(error)
        }
    }

    private func handleFrame(_ sampleBuffer: CMSampleBuffer) {
        rollingFrameBufferQueue.async { [weak self] in
            guard let self = self else { return }
            let pts = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
            var retainedBuffer: CMSampleBuffer?
            CMSampleBufferCreateCopy(allocator: kCFAllocatorDefault, sampleBuffer: sampleBuffer, sampleBufferOut: &retainedBuffer)
            guard let buffer = retainedBuffer else {
                print("[DEBUG] Failed to copy sample buffer")
                return
            }
            self.rollingFrameBuffer.append((buffer, pts))
            if let lastPTS = self.rollingFrameBuffer.last?.1 {
                while let firstPTS = self.rollingFrameBuffer.first?.1,
                      CMTimeGetSeconds(CMTimeSubtract(lastPTS, firstPTS)) > self.rollingBufferDuration {
                    self.rollingFrameBuffer.removeFirst()
                }
            }
        }
    }
    
    func stream(_ stream: SCStream, didStopWithError error: Error) {
        print("[DEBUG] Stream stopped with error: \(error.localizedDescription)")
        displayCheckQueue.async { [weak self] in // Move to displayCheckQueue for synchronization
            self?.handleStreamError(error)
        }
    }
    
    

what could be the reason for this and what would be the possible fix logically? i dont understand why it's dependant on queueDepth, and if it is, how can I empty and append new recorded frames to it so that it continues working? any help or resource is greatly appreciated!

what type of object is a ScreenStreamOutput? I don't see it anywhere in Apple's documentation.

I'd expect your app, or a component thereof, to conform to SCStreamOutput protocol. So it should implement a

func stream(
    _ stream: SCStream,
    didOutputSampleBuffer sampleBuffer: CMSampleBuffer,
    of type: SCStreamOutputType
)

that's where you'd put logic which copies the content of the sampleBuffer into your private circular buffer.

thank you for that @ssmith_c ! i realised that I didn't have that structure protocol setup that way. I now can record frames properly with queueDepth = 8.

However, there is another issue I run into (which I don't know how to reproduce or trigger it)

My stream suddenly breaks/stops sometimes and I get the error: EXC_BAD_ACCESS (code=1, address=0x0)

I catch it in this function:

func stream(_ stream: SCStream, didStopWithError error: Error) {
        print("Stream stopped with error: \(error.localizedDescription)")
        if isRecording {
            isRecording = false
            if let session = compressionSession {
                VTCompressionSessionCompleteFrames(session, untilPresentationTimeStamp: .positiveInfinity)
                VTCompressionSessionInvalidate(session)
            }
            compressionSession = nil
            self.stream = nil
            self.bufferQueue.async {
                print("[DEBUG] Buffer size before restart: \(self.circularBuffer.count)")
            }
            // Attempt to restart
            self.start()
            self.bufferQueue.async {
                print("[DEBUG] Buffer size after restart: \(self.circularBuffer.count)")
            }
        }
    }

one moment, the stream would be running properly and then the next, it stops.

What are generally the steps to debug in this scenario? I added exception breakpoints but I don't exactly know what to look for and where to look for. I added zombie objects detection as well but I don't know what to check for..

some code accessed a nil pointer. The code doesn't belong to your process (or else you would have landed in the debugger with a BAD_ACCESS error), instead didStop is called. However, almost undoubtedly your code caused the nil pointer access.

Take a close look at how you manage video buffers and read the documentation very carefully. CMSampleBufferCreateCopy doesn't actually copy the buffer's data, for example.

ScreenCapture + CMSampleBuffer logic issue
 
 
Q