Xcode 13.4 seems to solve this issue!

Did anyone check if Xcode 13.4 RC solves this?

Same happens for me since Xcode 13.3 and even on Xcode 13.3.1. The only workaround is to wait like 2 hours for the build to actually finish.

Do you happen to have an anti-virus on your mac? Falcon (crowdstrike), bit-defender?

Happens for me as well since Xcode 13.3