I seem to have fixed mine by turning off the auto switching. I set connect to this mac to "when last connected to this mac" and did the same on my phone. It seems to work, but time will tell.