Switching ssh keys from ecdsa-sha2 to rsa on that git user fixed the issue for me.

Thank you all above for posting so much detail. In the end it was exactly the fact that it was not a RSA key that fixed the issue for me. I rotated out a ecdsa-sha2 key for rsa and it worked. Of course in doing so I updated the server authorized keys as well, so Xcode just likes a very particular setup.