Instead of restarting your Mac just turn the mouse on and off, its a much quicker way to get past this. I have MB Air M1 and its been doing it for awhile now.