Same issue here. It happened when upgrading to Catalina a couple of years ago. The issue disappeared after several upgrades though. Now it is back again on Big Sur :(

I can confirm the fix works for me, thanks for the great tip! System Preferences -> Display -> HDMI Monitor -> Color -> Calibrate... -> Click through all the dialogs, Done!