I'm facing the same issue and still didn't find any solution. Went through all replies but didn't help.

Experiencing same bug in my new iPhone SE (Just Purchased), installed with latest iOS version 14.6 Whereas in my old iPhone 6S, it's working fine. Apple must fix this issue, as its really annoying. I've added location permission, which I accepted and now I can not change it. App installed directly from Xcode and tried also with TestFlight. Tried everything related to device, like killing the app and fresh start again. Device turn off and on again.