RealityKit crashes when rendering SpriteKit scene with SKShapeNode in postProcess callback

I'm converting my game from SceneKit to RealityKit. It has a SpriteKit overlay that according to Explore advanced rendering with RealityKit 2 I can add with the code below.

The code runs fine if the SKScene only contains a SKSpriteNode (see the commented out line), but when I add a SKShapeNode with a fillColor instead, the app crashes with this error:

-[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5970: failed assertion `Draw Errors Validation
MTLDepthStencilDescriptor uses frontFaceStencil but MTLRenderPassDescriptor has a nil stencilAttachment texture
MTLDepthStencilDescriptor uses backFaceStencil but MTLRenderPassDescriptor has a nil stencilAttachment texture
'

I don't know enough about low-level graphics and stencils yet to figure out a quick solution, so I would appreciate if anyone could share an easy fix or explanation of what's wrong. Thanks!

class ViewController: NSViewController {
    
    var device: MTLDevice!
    var renderer: SKRenderer!
    
    override func loadView() {
        let arView = ARView(frame: NSScreen.main!.frame)
        view = arView
        
        arView.renderCallbacks.prepareWithDevice = { [weak self] device in
            guard let self = self else { return }
            self.device = device
            renderer = SKRenderer(device: MTLCreateSystemDefaultDevice()!)
            let scene = SKScene()
            let shape = SKShapeNode(rectOf: CGSize(width: 10, height: 10))
            shape.fillColor = .red
            scene.addChild(shape)
//            scene.addChild(SKSpriteNode(color: .red, size: CGSize(width: 10, height: 10)))
            renderer.scene = scene
        }
        arView.renderCallbacks.postProcess = { [weak self] context in
            guard let self = self else { return }
            let encoder = context.commandBuffer.makeBlitCommandEncoder()
            encoder?.copy(from: context.sourceColorTexture, to: context.targetColorTexture)
            encoder?.endEncoding()
            renderer.update(atTime: context.time)
            let descriptor = MTLRenderPassDescriptor()
            descriptor.colorAttachments[0].loadAction = .load
            descriptor.colorAttachments[0].storeAction = .store
            descriptor.colorAttachments[0].texture = context.targetColorTexture
            renderer.render(withViewport: CGRect(x: 0, y: 0, width: context.targetColorTexture.width, height: context.targetColorTexture.height), commandBuffer: context.commandBuffer, renderPassDescriptor: descriptor)
        }
    }
    
}

This reminds of an error I got while setting up SKRenderer for this project: SKRenderer Demo.

See code and comment in SKOfflineRenderer.swift:

// If I dont use a depth/stencil texture, rendering crashes on simulator, device, and Mac
// Without it, rendering only works on Xcode Live Preview
let depthStencilDesc = MTLTextureDescriptor()
depthStencilDesc.pixelFormat = .depth32Float_stencil8
depthStencilDesc.width = pixelWidth
depthStencilDesc.height = pixelHeight
depthStencilDesc.usage = .renderTarget
depthStencilDesc.storageMode = .private

The code is inside the init, check how the depthStencilTexture is setup for SKRenderer, it may help you.

Thanks for your input. I added the following code in the arView.renderCallbacks.postProcess callback:

let depthStencilDescriptor = MTLTextureDescriptor()
depthStencilDescriptor.pixelFormat = .depth32Float_stencil8
depthStencilDescriptor.width = context.targetColorTexture.width
depthStencilDescriptor.height = context.targetColorTexture.height
depthStencilDescriptor.usage = .renderTarget
depthStencilDescriptor.storageMode = .private
renderPassDescriptor.stencilAttachment.texture = device!.makeTexture(descriptor: depthStencilDescriptor)
renderPassDescriptor.stencilAttachment.loadAction = .clear
renderPassDescriptor.stencilAttachment.storeAction = .dontCare
renderPassDescriptor.stencilAttachment.clearStencil = 0

but the app now crashes with error

Assertion failed: (destDepthFormat == jet_texture_format_DepthStencil), function create_render_mode, file jet_context_Metal.mm, line 895.
VTPixelTransferSession  420f sid 494 (512.00 x 512.00) [0.00 0.00 512 512] rowbytes( 512, 512 ) Color( kCGColorSpaceSRGB, 0x0, (null), (null), ITU_R_601_4 ) => BGRA sid 554 (512.00 x 512.00) [0.00 0.00 512 512] rowbytes( 2048 ) Color( 0x0, (null), (null), (null) )

which again I have no idea what it means.

Ache's reply pointed you in the right direction. SKShapeNode uses stencil operations internally to render filled shapes, so the render pass descriptor needs a stencil attachment. The reason your fix attempt crashed with destDepthFormat == jet_texture_format_DepthStencil is that you only set the stencil attachment — with a .depth32Float_stencil8 combined format, the same texture must be assigned to both depthAttachment and stencilAttachment.

Here's the corrected setup. Create the depth/stencil texture once (not every frame), then assign it to both attachments:

// Create once during prepareWithDevice, alongside your SKRenderer
let depthStencilDesc = MTLTextureDescriptor()
depthStencilDesc.pixelFormat = .depth32Float_stencil8
depthStencilDesc.width = /* target width */
depthStencilDesc.height = /* target height */
depthStencilDesc.usage = .renderTarget
depthStencilDesc.storageMode = .private
let depthStencilTexture = device.makeTexture(descriptor: depthStencilDesc)

Then in your postProcess callback, attach it to the render pass descriptor:

let descriptor = MTLRenderPassDescriptor()
descriptor.colorAttachments[0].loadAction = .load
descriptor.colorAttachments[0].storeAction = .store
descriptor.colorAttachments[0].texture = context.targetColorTexture

descriptor.depthAttachment.texture = depthStencilTexture
descriptor.depthAttachment.loadAction = .clear
descriptor.depthAttachment.storeAction = .dontCare
descriptor.depthAttachment.clearDepth = 1.0

descriptor.stencilAttachment.texture = depthStencilTexture
descriptor.stencilAttachment.loadAction = .clear
descriptor.stencilAttachment.storeAction = .dontCare
descriptor.stencilAttachment.clearStencil = 0

One other thing: in your prepareWithDevice callback, you're creating the SKRenderer with MTLCreateSystemDefaultDevice() instead of the device parameter that RealityKit passes to you. Use the provided device to ensure both renderers share the same Metal device.

Thanks for your help. Now the app doesn't crash anymore, but the colours of textures shown in SKSpriteNode are a little bit off.

To reproduce this, add the below image as a new image resource named "blue" in the Xcode project. It contains a single color RGB = (71, 147, 191).

When shown in ARView, it is rendered as RGB = (93, 201, 227), which is visibly different than the original image.

When shown in SKView, it is rendered as RGB = (28, 149, 195), which is still slightly different than the original image, but close enough that I didn't notice it before doing this measurement.

Why is the colour so different in ARView? (Optional question: why is it also slightly different in SKView?)

Also, I'm not sure what I should do in order to not create the texture every frame, since its size is dependent on the size of the postProcess context. Should I store the texture in an instance variable and generate it again when the size of the old one doesn't match the size of context.targetColorTexture?

I created FB22541137.

import Cocoa
import SpriteKit
import RealityKit

class ViewController: NSViewController {
    
    var device: MTLDevice!
    var renderer: SKRenderer!
    
    override func loadView() {
        let arView = ARView(frame: NSScreen.main!.frame)
        view = arView
//        let skView = SKView(frame: NSScreen.main!.frame)
//        skView.presentScene(createScene())
//        view = skView
        
        arView.renderCallbacks.prepareWithDevice = { [weak self] device in
            guard let self = self else { return }
            self.device = device
            renderer = SKRenderer(device: device)
            renderer.scene = createScene()
        }
        arView.renderCallbacks.postProcess = { [weak self] context in
            guard let self = self else { return }
            let encoder = context.commandBuffer.makeBlitCommandEncoder()
            encoder?.copy(from: context.sourceColorTexture, to: context.targetColorTexture)
            encoder?.endEncoding()
            renderer.update(atTime: context.time)
            let depthStencilDescriptor = MTLTextureDescriptor()
            depthStencilDescriptor.pixelFormat = .depth32Float_stencil8
            depthStencilDescriptor.width = context.targetColorTexture.width
            depthStencilDescriptor.height = context.targetColorTexture.height
            depthStencilDescriptor.usage = .renderTarget
            depthStencilDescriptor.storageMode = .private
            let texture = device!.makeTexture(descriptor: depthStencilDescriptor)
            let renderPassDescriptor = MTLRenderPassDescriptor()
            renderPassDescriptor.colorAttachments[0].loadAction = .load
            renderPassDescriptor.colorAttachments[0].storeAction = .store
            renderPassDescriptor.colorAttachments[0].texture = context.targetColorTexture
            renderPassDescriptor.depthAttachment.texture = texture
            renderPassDescriptor.depthAttachment.loadAction = .clear
            renderPassDescriptor.depthAttachment.storeAction = .dontCare
            renderPassDescriptor.depthAttachment.clearDepth = 1.0
            renderPassDescriptor.stencilAttachment.texture = texture
            renderPassDescriptor.stencilAttachment.loadAction = .clear
            renderPassDescriptor.stencilAttachment.storeAction = .dontCare
            renderPassDescriptor.stencilAttachment.clearStencil = 0
            renderer.render(withViewport: CGRect(x: 0, y: 0, width: context.targetColorTexture.width, height: context.targetColorTexture.height), commandBuffer: context.commandBuffer, renderPassDescriptor: renderPassDescriptor)
        }
    }
    
    private func createScene() -> SKScene {
        let scene = SKScene(size: CGSize(width: 100, height: 100))
        scene.addChild(SKSpriteNode(texture: SKTexture(imageNamed: "blue"), size: CGSize(width: 100, height: 100)))
        return scene
    }
    
}
RealityKit crashes when rendering SpriteKit scene with SKShapeNode in postProcess callback
 
 
Q