I ditched this approach completely and went with a solution that uses scheduled push notifications to get location updates

https://stackoverflow.com/questions/77773014/change-swiftui-map-position-without-resetting-distance

The best way I found for this was to send scheduled background push notifications from a server and update location when those are received.

I am also using a CLBackgroundActivitySession

Same issue here