Changing Frame Rate of External Display on iPad

Hey @bradley_7,

I thought it was impossible to change the video output frame rate

It was impossible, until iOS 26.1 introduced AVCaptureExternalDisplayConfigurator :)

AVCaptureExternalDisplayConfigurator configures properties of a connected external display to match the camera's active video format. It enables the external display to output a clean feed using a CALayer.

Using the configurator, you can enable automatic adjustments of the external display's color space and/or frame rate to synchronize with the device's capture configuration. These adjustments apply exclusively to the external display, not to the capture device itself.

After you read over the docs, let me know if you have any further questions about it!

--Greg

Thank you for the answer and explanation Greg, much appreciated.

The 'init' for 'AVCaptureExternalDisplayConfigurator' refers to the CALayer as 'previewLayer'. Does it work on any CALayer or does it need to specifically be a need to be a 'AVCaptureVideoPreviewLayer'?

I tried to use it on a CALayer like this:

    func setupDisplayConfigurator() {
        if #available(iOS 26.0, *) {
            // Fetch Video Input
            guard let videoInput = videoInput else { return }
            
            // Fetch CALayer
            for window in additionalWindows {
                print("window: \(window) | \(window.rootViewController?.view.layer)")
            }
            guard let layer = additionalWindows.first?.rootViewController?.view.layer else { return }
            
            // Create Display Config
            let config = AVCaptureExternalDisplayConfiguration()
            config.shouldMatchFrameRate = true
            
            // Create Configurator
            let configurator = AVCaptureExternalDisplayConfigurator(device: videoInput, previewLayer: layer, configuration: config)
        }
    }

The CALayer is coming from an array of UIWindow:

    private var additionalWindows: [UIWindow] = []

When I call it like this the app crashes:

*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[AVCaptureExternalDisplayConfigurator initWithDevice:previewLayer:configuration:] shouldMatchFrameRate is not supported'

It's unclear why this crash is occuring, is it because the CALayer is not a 'AVCaptureVideoPreviewLayer'? I know the external display can handle UHD 60fps, I am feeding it HD at 60fps, my intention is to change this 25fps which matches the frame rate of the External UVC Input Device.

Thank you!

Also, in the past I have done extensive testing with 'AVCaptureVideoPreviewLayer'. My understanding is that you can only ever have one 'AVCaptureVideoPreviewLayer' in use, is that right?

In my use case the Live Video from a UVC Capture Card (AVCaptureDevice) needs to be displayed on the iPad Screen as well as the External Display.

There seems to be something special about 'AVCaptureVideoPreviewLayer' as it has the least delay from Capture Device to the Screen. When I try to implement any other way of displaying the image from the CMSampleBuffer there is always more delay than using 'AVCaptureVideoPreviewLayer' directly. Around 1fps is the delay from my measurements.

I ended up ditching 'AVCaptureVideoPreviewLayer' due to this dual use limitation and moving a to fully Metal backed pipeline. The CMSampleBuffer is converted to a MTLTexture and this is used for the iPad and the External Display. This also allows for other GPU Backed Functionality like LUTs, False Color, Focus Peaking, etc. The only trade off is the slight increase in delay.

Given this new functionality with AVCaptureExternalDisplayConfigurator, my initial understanding is that it wouldn't be able to be used with a MTLTexture? Only a CALayer? Could you pass the CALayer from the MTKView into this to match the frame rate?

Also I have recently implemented support for a MFi Device that brings in H264 Video Frames directly, so if a user was using that instead of a UVC Capture Card (AVCaptureDevice) then this framework wouldn't be available right?

I haven't got to playback yet, in the app the user can also playback video files that were recorded from the AVCaptureDevice or MFi Device, if the recorded file is 25fps, I couldn't use this API to set the external display to be at 25fps as it's only for Live Video, is that correct?

Hey @bradley_7,

The 'init' for 'AVCaptureExternalDisplayConfigurator' refers to the CALayer as 'previewLayer'. Does it work on any CALayer or does it need to specifically be a need to be a 'AVCaptureVideoPreviewLayer'?

As noted in the previewLayer property docs:

"You may specify either an AVCaptureVideoPreviewLayer or another CALayer instance that displays a camera’s video preview."

When I call it like this the app crashes:

*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[AVCaptureExternalDisplayConfigurator initWithDevice:previewLayer:configuration:] shouldMatchFrameRate is not supported'

It's unclear why this crash is occuring, is it because the CALayer is not a 'AVCaptureVideoPreviewLayer'?

That exception is being thrown because isMatchingFrameRateSupported is reporting false for your external display, but your AVCaptureExternalDisplayConfiguration specified true for shouldMatchFrameRate.

Also, in the past I have done extensive testing with 'AVCaptureVideoPreviewLayer'. My understanding is that you can only ever have one 'AVCaptureVideoPreviewLayer' in use, is that right?

That is correct.

Given this new functionality with AVCaptureExternalDisplayConfigurator, my initial understanding is that it wouldn't be able to be used with a MTLTexture? Only a CALayer? Could you pass the CALayer from the MTKView into this to match the frame rate?

You should be able to access and use the CAMetalLayer of the MTKView for this purpose, though I haven't tried it myself.

Also I have recently implemented support for a MFi Device that brings in H264 Video Frames directly, so if a user was using that instead of a UVC Capture Card (AVCaptureDevice) then this framework wouldn't be available right?

I haven't got to playback yet, in the app the user can also playback video files that were recorded from the AVCaptureDevice or MFi Device, if the recorded file is 25fps, I couldn't use this API to set the external display to be at 25fps as it's only for Live Video, is that correct?

Right, AVCaptureExternalDisplayConfigurator requires an AVCaptureDevice.

--Greg

Thank you for all of that information Greg and for answering each query individually.

That exception is being thrown because isMatchingFrameRateSupported is reporting false for your external display, but your AVCaptureExternalDisplayConfiguration specified true for shouldMatchFrameRate.

What determines 'isMatchingFrameRateSupported'? I use the same USB Hub and External Display on a MacBook Pro and I can go to Settings -> Displays and change the resolution and frame rate. I tested 1920 x 1080 at 60fps, 50fps, 30fps and 25fps and they all work. When I use that exact same hardware on an M4 iPad Pro it returns 'false' for 'isMatchingFrameRateSupported', presumably iPadOS is limiting this?

Great to know that I can use CAMetalLayer for this purpose. That would mean it could work for a Metal Backed Live Video Feed if I can get the display to work as per the above.

Though I think given the limitation of this functionality to AVCaptureDevice and that it wouldn't be respected when the user plays back video from within the app it may not be worth implementing.

I feel like it's about time iPadOS opened up frame rate selection for external displays to the user. It's getting more and more like a Desktop OS each year and many people use their iPad with an external display. Sure, have it default to what iPadOS thinks is best, but having an option in settings to set the resolution and frame rate doesn't feel like that big of an ask for users that know what they need for their specific use case.