Xcode behave differently on SPM between Package project and iOS project.

I'm now developing a iOS project and a SPM package project, which both depend on another swift package stored in a private repo.

Our private repo has a non-default ssh port, so for commandline tools, I configured like this:

// .ssh/config
Host repo.private.com
User git
Port 20022
IdentityFile ~/.ssh/id_rsa
PreferredAuthentications publickey
AddKeysToAgent yes

And I modified defaults to disable xcode integrated git:

defaults write com.apple.dt.Xcode IDEPackageSupportUseBuiltinSCM YES

For the Package project, I can simply use:

.package(url: "git@repo.private.com:some-repo.git", from: "0.0.1"),

and it works well.

But for iOS project, I tried to add the dependency, and finally find out I have to configure the port, then the project packages shows as: ssh://git@repo.private.com:20022/some-repo.git

So does xcode treat these project differently and use different toolchains? Is it possible to configure xcode and make these situation behave exactly?