I'm having the same issue here in 2025. I have to download and install Swift packages locally in each project because the ssh handshake fails. I work on enterprise apps so this is beyond tedious.