I'm still dealing with exactly the same problem, so for now I have decided to use the UIViewRepresentable road. Let me know if something improves with the SwiftUI Map.