I was having this issue with a Logitech M720 mouse and enabling Logi Options in Accessibility settings fixed that without a reboot. Not sure if that helps in any way with the Magic Mouse though.