Thanks for the idea. I did that, but it did not change my problem. I still was able to connect via terminal but not via xcode. But it got me thinking that maybe something is wrong with the key. I created a new ssh key and with this it works. It seems that xcode does not like rsa keys. Using an ed25519 key solved it.