As explained a few days ago on Reddit, it seems that Intel processes which are "non-native" for Macs running on M1 processors seem to be the culprit here.
Check the Activity Monitor for any processes in the CPU view listed as type "Intel". The type column might be hidden, so you need to unhide it. I've noticed that if I avoid running any applications which spawn Intel type processes, I don't get any distortions or issues. However, even music players like Qobuz still run as an Intel process, and one or two processes is generally fine. However I get too many of those Intel processes running, the popping etc starts happening immediately.
I guess it might be something to do with how the Rosetta 2 translator handles those kind of processes and too much CPU and/or memory consumption starts causing problems.
I have updated Java, node etc. to run natively as Apple processes. Unfortunately many apps I need for work still run non-natively on M1 Macs and thus causing problems with audio.