I had this same problem and found the solution over at the Swift Forums: https://forums.swift.org/t/xcode-and-swift-package-manager/44704
The TLDR is that the Swift PM cache directory needs to be outside of your main project directory and using -derivedDataPath messes that up. You need to use the -clonedSourcePackagesDirPath parameter as well to point to a directory outside of your project folder to fix it.
Topic:
Developer Tools & Services
SubTopic:
Xcode
Tags: