LowLevelRenderer: visionOS custom shaders, composition with RealityView, custom surface shader ABI

Question

Created 4h

Replies 2

Boosts 3

Participants 2

This post is from the WWDC26 RealityKit & Reality Composer Pro Q&A.

A few questions about the intended usage of LowLevelRenderer / LowLevelRenderContextStandalone for custom rendering (custom Metal materials, dynamic meshes, custom splat compositing):

visionOS custom shaders in immersive space

LowLevelRenderer is listed as available on visionOS 27. Does that include user-authored Metal surface shaders via makeSurfaceShader when rendering in an immersive space, or are custom fragment shaders restricted there the way CustomMaterial was? Any foveation / Compositor Services constraints we should know about?

Composition with RealityView

Is LowLevelRenderer (via LowLevelRenderContextStandalone) intended only for fully standalone/offscreen rendering where the caller composites the output, or can it inject draws into an existing RealityView's managed render pass? On visionOS specifically, what's the supported way to get its output onto the drawable alongside RealityKit-rendered content with correct depth interaction (mutual occlusion between custom draws and normal entities)?

Custom surface shader ABI

For makeSurfaceShader with a user-authored Metal function: what's the function signature contract? Specifically — which [[buffer(n)]] / [[texture(n)]] slots are reserved by the renderer vs. available to the caller, how do camera and per-instance uniforms arrive in the shader, and how do LowLevelArgumentTable bindings map to argument indices? A pointer to sample code or header documentation for the shader-authoring contract would be ideal.

Hot reload / resource lifetime

For an authoring tool that recompiles shaders frequently: can LowLevelRenderContextStandalone.Resources be rebuilt incrementally, and what's the cost of re-init via the synchronous init(resources:) path? Any guidance on reusing resources across context instances?

Relationship to GaussianSplatComponent

Is GaussianSplatComponent built on LowLevelRenderer internally? For 4D / animated splats and counts above the component's internal limit, is implementing custom splat rendering directly on LowLevelRenderer a path you'd recommend, or discourage?

Thank you!

Answered by Vision Pro Engineer in 891598022

User-authored Metal surface shaders are supported with LowLevelRenderer on visionOS. Both stereo rendering and rasterization rate maps are supported features on LowLevelRenderer.output, so interoperability with CompositorServices API is an intended use case.

LowLevelRenderer / LowLevelRenderContextStandalone is intended for use in offscreen draws / exclusive rendering use cases such as through CompositorServices. Composition with RealityView content can be done by presenting LowLevelRenderer's output to a LowLevelTexture, sampled from a RealityKit mesh via ShaderGraphMaterial. Depth compositing can also be approximated by using a tesselated grid mesh, again through ShaderGraphMaterial and geometry modifier.

I will get back to you regarding the remaining questions.

As always, please do not hesitate to file feedback for any requests and/or issues, for any use cases you are interested in pursuing.

Answer 1

Vision Pro Engineer OP

Apple

3h

Recommended

User-authored Metal surface shaders are supported with LowLevelRenderer on visionOS. Both stereo rendering and rasterization rate maps are supported features on LowLevelRenderer.output, so interoperability with CompositorServices API is an intended use case.

LowLevelRenderer / LowLevelRenderContextStandalone is intended for use in offscreen draws / exclusive rendering use cases such as through CompositorServices. Composition with RealityView content can be done by presenting LowLevelRenderer's output to a LowLevelTexture, sampled from a RealityKit mesh via ShaderGraphMaterial. Depth compositing can also be approximated by using a tesselated grid mesh, again through ShaderGraphMaterial and geometry modifier.

I will get back to you regarding the remaining questions.

As always, please do not hesitate to file feedback for any requests and/or issues, for any use cases you are interested in pursuing.

Answer 2

arthurfromberlin OP

1h

@Vision Pro Engineer This fully answers the architecture question — thank you. Custom Metal surface shaders on visionOS with foveation + CompositorServices interop is exactly what we needed to hear, and the LowLevelTexture → ShaderGraphMaterial route (with the tessellated-grid geometry-modifier trick for approximate depth) is clear.

The one thing that'd be transformative for us: a first-class way to render LowLevelRenderer content inside an existing RealityKit scene with shared depth — true mutual occlusion against normal entities, rather than texture-on-a-mesh with approximated depth. Filed as a feature request, FB23042990 — essentially a "LowLevelRendererComponent" or a render-callback hook on RealityView. Our use case is mixing custom-rendered content (splats, custom Metal materials, dynamic meshes) with standard entities in one coherent scene on visionOS.

Still very interested in the surface-shader ABI follow-up when you have it — the function signature contract (reserved vs. available buffer/texture slots, how camera/instance uniforms arrive, how LowLevelArgumentTable maps to argument indices), ideally with a pointer to sample code. Thanks again!