Yes, I have the same (currently on my M4 Pro, previously on my M1 Pro). Specifically noticeable if you are running music in the background while working in Xcode. The 'solution' I have is to kill the audio daemon. It will then auto-restart and it's ok for at least a while. Use: 'sudo killall coreaudiod'.