upd: I didn't work

I'm not sure now, since the project died 4 months ago anyway. But I still had the same issue at that time. Now I'm working with a project, that doesn't have that complicated structure for now, and I'm not facing these issues anymore. Looks like the more local Swift Packages and project targets you have, the more terrible Xcode works. Sorry for the late answer, I missed all the replies somehow, although I'm subscribed to this thread.