I was hitting this problem and found Xcode kept trying to connect to other devices that it had used previously, even though I wasn't current debugging with them.
eg, it would try and connect to my personal phone and watch over the wifi network even though I wasn't using them and I had the test device connected via usb. As a workaround i tried two things (not sure about the second one, but it all seems pretty good at the moment).
First thing I think helps the most was go into Window | Devices And Simulators and set the Run Destination for all devices to Never and then go back to just the devices u want to debug to Auto or Always - hopefully this will stop Xcode trying to connect simultaneously to anything it has ever used.
I've also got both ethernet and WIFI running (can't get air drop to work over ethernet, despite Apple saying iOS 17 it does work, and the ethernet interface is much faster generally). So I've changed the priority of the services in System | Network so that ethernet is above WIFI
These two changes seem to help a lot and for me it's back to what it used to be