Blocking network access also sped up my xcodebuild builds a lot on XCode 16.2. I used LuLu, an outbound firewall app, to selectively block network access from xcodebuild.

This was helpful, I appreciate all the followup responses, thanks