In case anyone else is following along, this issue is still present in Xcode 15 Beta 3! https://developer.apple.com/documentation/xcode-release-notes/xcode-15-release-notes#visionOS-Simulator

I guess it depends on whether you're going to be doing RealityKit shaders early on. Aside from this issue, the simulator is running ok on my Intel iMac, and it lets me poke around with SwiftUI stuff etc. As long as I'm not trying to do fancy RealityKit shaders, it's usable. And, I have hope that a fix will be forthcoming soon!