In my original case I'm just hitting Run ▶️ in Xcode, visionOS simulator not running and I get this. Simulator starts though. I can then go into the simulator and find my app and run it.

No, I didn't figure it out. I was trying to create a xcframework with both arm64 and arm64-simulator in one go, but I instead went the route of creating separate framework for each slice and then combining them with xcodebuild -create-xcframework -framework <framework1> -framework <framework2> and this worked for me.