Same sort of issues here. Mac Mini M1, running 2 x 27 inch displays- one connected via HDMI directly, and another connected via USB-C to HDMI adapter. Every time upon a rewake from sleep/ restart, I will need to redo all the display settings: rearrange monitors so that my mouse scrolls properly across both screens adjust scaling on the 4k monitor to display 2560x1440, as it would default to 1920x1080 Super irritating! I tried using another USB-C to HDMI adapter....same issue. Tried the calibration, NVRAM reset and SMC reset as suggested by some in this thread. Didn't work either.

Update: Somehow after unplugging the HDMI cables from the Mac Mini M1, from the HDMI port and the USB-C to HDMI adapter, and swapping them, around, I was able to remedy the situation. I no longer need to go into display settings to rescale/ position the monitors after restart/ sleep. Try swapping the ports /cables if your situation persists. I figure this is the best way until Apple releases a fix.

Oops my fix didn't work overnight. It reverted to the same issue again!

fixed with the latest Big Sur 1.1 for me. M1 mini with 2 x 27 inch displays (1 x HDMI, 1 using USB-C to HDMI adapter), running well now and no longer randomly messing up the display order and scaling after restart/ rewake from sleep.