EnvironmentBlendingComponent(.occluded(by: .surroundings)) culling entities entirely?

Hi there!

While building my visionOS app, I’ve encountered some strange behaviour with EnvironmentBlendingComponent. Entities using .occluded(by: .surroundings) are culled entirely whenever a Mac Virtual Display or Apple Immersive Environment is behind them, regardless of the entities’ actual depth, and even when the environment is set to coexist.

This feels like it could be a bug, although I’m fairly new to visionOS development, so I may be missing something!

In a .mixed immersive space with plain passthrough, the component behaves as I would expect: an entity is occluded accurately when a real-world object is placed in front of it.

However, when either of the following is visible behind the entity:

  • a Mac Virtual Display window; or
  • an Apple Immersive Environment enabled using the Digital Crown,

the entity carrying EnvironmentBlendingComponent disappears completely. This happens even when the entity is physically closer to the viewer than the virtual surface. An otherwise identical entity without the component remains visible.

Moving the virtual surface out of the line of sight, disabling the Immersive Environment, or removing the component immediately makes the entity visible again.

Is it intended that .occluded(by: .surroundings) treats Mac Virtual Display and Apple Immersive Environments as occluders? The documentation describes this mode as depth-based occlusion against static “real-world objects”, so it isn’t clear whether these system-rendered virtual surfaces should participate.

Even if they are intended to participate, the observed behaviour appears depth-independent: the entire entity is hidden when the virtual surface is behind it, rather than only when the surface is closer to the viewer. That seems inconsistent with the documented depth-based behaviour.

I can reproduce this on an M5 Apple Vision Pro running visionOS 26.5 using the following minimal example:

import SwiftUI
import RealityKit
import UIKit

@main
struct OcclusionReproApp: App {
    var body: some SwiftUI.Scene {
        WindowGroup {
            OcclusionReproLauncher()
        }

        ImmersiveSpace(id: "immersive") {
            RealityView { content in
                // Control: opaque, no component — proves the bare setup renders.
                let control = ModelEntity(
                    mesh: .generateSphere(radius: 0.12),
                    materials: [SimpleMaterial(color: .green, isMetallic: false)]
                )
                control.position = [-0.25, 1.2, -1.3]   // ~1.3 m ahead, eye height

                // Under test: identical opaque sphere plus the occlusion component.
                let occluded = ModelEntity(
                    mesh: .generateSphere(radius: 0.12),
                    materials: [SimpleMaterial(color: .red, isMetallic: false)]
                )
                occluded.position = [0.25, 1.2, -1.3]
                occluded.components.set(
                    EnvironmentBlendingComponent(preferredBlendingMode: .occluded(by: .surroundings))
                )

                content.add(control)
                content.add(occluded)
            }
        }
        .immersionStyle(selection: .constant(.mixed), in: .mixed)
        // Let the app's mixed space coexist with an Apple Immersive Environment,
        // as visionOS suppresses Environments while an immersive space is open otherwise.
        // The same bug can be reproduced with just a Mac Virtual Display, but
        // this provides a second way to reproduce the same behaviour:
        .immersiveEnvironmentBehavior(.coexist)
    }
}

struct OcclusionReproLauncher: View {
    @Environment(\.openImmersiveSpace) private var openImmersiveSpace

    var body: some View {
        Button("Open immersive space") {
            Task { await openImmersiveSpace(id: "immersive") }
        }
        .padding()
    }
}

To reproduce (on a Vision Pro, as the Simulator won't provide passthrough):

  1. Open the immersive space in plain passthrough. Both spheres should be visible.
  2. Place a real object in front of the red sphere. It should be occluded correctly according to depth.
  3. Open a Mac Virtual Display or enable an Apple Immersive Environment, with its visible surface behind the red sphere.
  4. The red sphere disappears entirely, despite being in front of that surface. The green control sphere remains visible.
  5. Move the virtual surface out of view or disable it, and the red sphere reappears.

Please let me know if this is expected behaviour, or if I'm doing something wrong - thanks!

Jack

Answered by Vision Pro Engineer in 894279022

Hi @jackwh,

Your understanding of the behavior of using SceneReconstructionProvider in tandem with OcclusionMaterial looks correct to me and would be a reasonable approach. See the Obscuring virtual items in a scene behind real world items sample for more information, if you haven't already.

As you've hinted at above, there is at least one potential drawback to using scene reconstruction and occlusion material in comparison to using environment blending component, which is that invisible/occluded reconstructed scene geometry will continue to occlude virtual content. For example, imagine that you have an object which is being partially occluded by your real-world environment, and you then move MVD behind that object or dial in a system environment. That object will render in front of said content, while continuing to be partially occluded by the real-world, leading to a disjointed visual state. EnvironmentBlendingComponent avoids this issue (by treating virtual objects with occlusion just like real-world passthrough objects), so it's worth considering the tradeoffs of each approach.

As for determining the immersion level of a system environment with .coexist, there's no supported way to do that with the APIs currently available as far as I'm aware.

Let me know if you have any further questions, thanks!

Hi @jackwh,

Thank you for the detailed question, what you're seeing is in fact the expected behavior.

This is briefly covered in the EnvironmentBlendingComponent documentation, though I can try to provide some additional context.

Regarding Mac Virtual Display:

An entity with occlusion enabled behaves like a real-world passthrough object. It always renders behind other virtual contents that don’t have occlusion enabled.

The key here is that entities with EnvironmentBlendingComponent(.occluded(by: .surroundings)) are occluded just like real-world objects. One way to think about this is that just like how all the real-world objects in your environment are occluded by Mac Virtual Display regardless of their "depth" (for example, the monitor on your desk, the coffee cup on your table, etc.), virtual entities with occlusion behave the exact same way.

Regarding system environments:

For progressive space, it works only for entities outside of the portal world. When the portal mask expands and overlaps with those entities by turning the Digital Crown, it fades those entities out gradually.

While the use of term "progressive space" is admittedly a bit confusing here, this is actually referring to exactly what you're seeing with immersiveEnvironmentBehavior(.coexist) where entities with occlusion become invisible when they overlap with the system environment made visible by turning the Digital Crown.

Thanks for such a comprehensive post—I really appreciate the detail (and the feedback ticket)!

Hey there, thank you so much - that's a really helpful and considered response, I appreciate it!

Just to double-check, before I go down this route instead: would the right approach here be to use SceneReconstructionProvider to get the physical environment mesh, render that with OcclusionMaterial, and leave my visible entities as normal virtual content without EnvironmentBlendingComponent?

My assumption is that this would give me the behaviour I was originally expecting:

  • real walls, furniture, etc. would occlude entities when they're genuinely in front
  • entities in front of real-world geometry would stay visible
  • entities would continue to depth-sort normally against Mac Virtual Display, and
  • they'd stay visible inside a system Immersive Environment when using .coexist, rather than being treated as part of the passthrough layer.

Does that sound like the right approach for this use case?

That would get me most of the way there, with one final part I'm unsure about: if I use the reconstructed physical mesh for occlusion, is there an API that lets my app detect how far the user has dialled in to system environments under .coexist?

For context, if an entity is positioned behind the user's sofa, I'd want it occluded while the sofa is visible in passthrough, but visible again once they dial into Bora Bora and the sofa is no longer part of their perceived environment.

I found onImmersionChange, but the docs are making me think this is exclusively for the app's immersion. If the system environment's immersion isn't exposed, is there any other way to avoid invisible reconstructed geometry continuing to occlude content?

Thanks once again, I really appreciate it! 🙏

Accepted Answer

Hi @jackwh,

Your understanding of the behavior of using SceneReconstructionProvider in tandem with OcclusionMaterial looks correct to me and would be a reasonable approach. See the Obscuring virtual items in a scene behind real world items sample for more information, if you haven't already.

As you've hinted at above, there is at least one potential drawback to using scene reconstruction and occlusion material in comparison to using environment blending component, which is that invisible/occluded reconstructed scene geometry will continue to occlude virtual content. For example, imagine that you have an object which is being partially occluded by your real-world environment, and you then move MVD behind that object or dial in a system environment. That object will render in front of said content, while continuing to be partially occluded by the real-world, leading to a disjointed visual state. EnvironmentBlendingComponent avoids this issue (by treating virtual objects with occlusion just like real-world passthrough objects), so it's worth considering the tradeoffs of each approach.

As for determining the immersion level of a system environment with .coexist, there's no supported way to do that with the APIs currently available as far as I'm aware.

Let me know if you have any further questions, thanks!

EnvironmentBlendingComponent(.occluded(by: .surroundings)) culling entities entirely?
 
 
Q