I have been struggling with this problem for a long time, but it seems to be completely fixed in macOS Sonoma 14.4 and Xcode 15.3. Everything works fine in my environment. Mac Studio 2022 (M1 Max)