We develop software for video broadcasting, including for sporting events, where animated scoreboards play a key role. macOS offers excellent features for such animations—such as CAAnimations—but this holds true only as long as the animations run on visible screens. Distributing the video signal (e.g., via NDI) requires access to the CVPixelBuffers of the individual frames. Currently, we generate the animations on an external screen and create the pixel buffers using ScreenCaptureKit. We are unaware of any way to obtain these pixel buffers without relying on such—strictly speaking, unnecessary—external screens (or multiple screens). If the window is created offscreen, it updates only once per second, which is unusable. Are there alternatives to external screens?
If not, why can’t we create a virtual offscreen device for such use cases—one where we specify the required frame rate and rendering frequency—to generate the necessary video frames?
This would also be helpful for HTML-based overlays, which are becoming increasingly popular; currently, these are also rendered only once per second when offscreen.
Consider using the CARenderer API instead of ScreenCaptureKit. CARenderer is an API for rendering a CALayer tree off the normal display path into a target you control, on a clock you drive.
You hand it a MTLTexture render target, and CoreAnimation via CARenderer will populate this texture. Media frameworks prefer CVPixelBuffers that are backed by IOSurfaces. You will make a MTLTexture that points to the data in a CVPixelBuffer (make sure it is IOSurface backed) with CVMetalTextureCache. Since this MTLTexture just points to the CVPixelBuffer, when CARenderer draws into it, the data will just be there in the CVPixelBuffer, which can then be handed over to Media frameworks like VTCompressionSession or AVFoundation to compress.
I would also highly encourage you to use CVPixelBufferPool. This is a mechanism for creating CVPixelBuffers and recycling resources. It is designed to work well with CVMetalTextureCache.