I have the same problem trying to build for VisionOS 2.2.

I found that using .handlesGameControllerEvents(matching: .gamepad) on the components that I don't want to steal focus fixed the issue.