This also occurs when using SwiftUI previews. I have the latest Big Sur 11.1(20C69) and Xcode 12.4 (12D4e)
For me killing/restarting the core audio daemon [$ sudo pkill coreaudiod] (scroll below) did the trick.
Still a pain... Apple please fix!
Topic:
App & System Services
SubTopic:
Core OS
Tags: