RealityKit .Kinematic + collisions (visionOs)

In order to control physical entities within a physics simulation, you can explore the topic of proportional-derivative (PD) controllers.

The principle:

• User input provides a target position
• Each frame (use the willSimulate event), a control system applies forces or impulses to drive the body toward that target

This answer provides a sample implementation with SpriteKit. The same principles apply to RealityKit.

Hello @SamApple , thank you for your question!

The interaction you are describing is a common one but it isn't obvious how to build it out of the box. However I think it will be very useful for developers to understand how to achieve "drag an object while maintaining solid collisions" in their app, so I spent some time working this out for myself. Keep in mind this is just one suggestion for how to implement this interaction, and there are likely edge cases that I won't cover here. Please feel free to ask any clarifying questions! I've also linked to the documentation pages on many of the component names so you can learn more about how each of these components work.

My implementation involves two entities: A target entity (a simple box in my example) which is what the person wants to manipulate and a "ghost" entity that the target will follow. The ghost will be the entity that is actually being manipulated, while the target will have forces applied to move it towards the ghost. Note that target entity remains .dynamic for the entire interaction. This is key to maintaining solid collisions with other entities in your scene.

1. Create the target entity and the ghost entity.

I made the ghost a descendant of the target entity and slightly bigger, so that the ghost is interacted with before the target. I setup the ghost using ManipulationComponent.configureEntity(_:) and gave it an OpacityComponent set to some small non-zero value so that it is hidden but still interactable.

2. Subscribe to ManipulationEvents.WillBegin

ManipulationEvents.WillBegin. When the manipulation begins, I unparent the ghost from the target so that the ghost and target are siblings. This will prevent their transforms from affecting each other. Additionally, I turn off gravity for the target entity and toggle a state flag on my View that I'll use later.

3. Subscribe to PhysicsSimulationEvents.WillSimulate

PhysicsSimulationEvents.WillSimulate. This is the bulk of the implementation so I'll just share the full function here:

func onPhysicsWillSimulate(event: PhysicsSimulationEvents.WillSimulate) {
    guard self.isManipulating else {
        return
    }
    
    guard let ghostBox = root.findEntity(named: ghostBoxName) else {
        return
    }
    
    guard let box = root.findEntity(named: boxName) else {
        return
    }
    
    // IMPORTANT! Convert the direction based on the orientation of the force effect entity.
    let toGhostBox = ghostBox.convert(direction: ghostBox.position - box.position, from: nil)
    let constantForceEffect = ConstantForceEffect(
        strength: 10,
        direction: toGhostBox
    )

    // Apply a drag force, otherwise things get crazy and start to orbit.
    let dragForceEffect = DragForceEffect(strength: 10)
    
    // Set the ForceEffectComponent on the ghost!
    ghostBox.components.set(ForceEffectComponent(effects: [
        ForceEffect(effect: constantForceEffect),
        ForceEffect(effect: dragForceEffect)
    ]))
}

Note that ForceEffectComponent acts like a "force field", which is why I'm able to apply it to the ghost entity rather than the target entity I actually want to move. Also note that root here is just some empty entity I created as a parent of everything that helps me find the specific entities I'm looking for. isManipulating is the state flag that I toggle on and off when the manipulation begins and ends.

4. Subscribe to ManipulationEvents.WillEnd

ManipulationEvents.WillEnd. I essentially reverse what I did in step 2: I reparent the ghost to the target entity, reset the ghost's transforms, and re-enable gravity for the target. I also remove the ForceEffectComponent from the ghost so that it no longer affects any physics objects in the scene.

I hope that's useful for you and anyone else reading! There's certainly more polish and tuning that could be done, and the values you use for the ForceEffects will depend on your app and the interaction you want to make.

EDIT: I realize the above solution does not allow for hover effects on the target entity, because the invisible ghost entity receives the gaze first. I think to solves this the core idea of having a target entity follow the ghost would remain the same, but there might need to be some refactoring to make the ghost be visible before the interaction (so it can show the hover effect) while the target is invisible, but then toggle opacity between the target and the ghost when the interaction starts. Let me know if you'd want me to go into that solution.

Let me know if you have any questions!