Interestingly enough - I am having the same problem with Big Sur. But I also had it on High Sierra since December or so.
It only seems to happen when I have external monitors connected. On High Sierra, changing from HDMI connection to Display Port connection solved the problem. But I upgraded to Big Sur the other day and now it is happening with both.
Model Name: iMac
Model Identifier: iMac19,1
Processor Name: 8-Core Intel Core i9
Processor Speed: 3.6 GHz
Number of Processors: 1
Total Number of Cores: 8
L2 Cache (per Core): 256 KB
L3 Cache: 16 MB
Hyper-Threading Technology: Enabled
Memory: 64 GB
System Firmware Version: 1554.80.3.0.0
SMC Version (system): 2.46f13
Topic:
App & System Services
SubTopic:
Core OS
Tags: