I found that using .handlesGameControllerEvents(matching: .gamepad) on the components that I don't want to steal focus fixed the issue.

I have the same problem trying to build for VisionOS 2.2.