Looks like the issues are all fixed in my case in Version 13.3 beta (13E5086k). They persisted eventhough Apple brought some 'intermediate' fix, not resolving all issues. I hope the fix is now persistent.

Turning off canvas prevents to run in this situation, i.e. background activities do not lither the memory and lead to the experienced misbehaviour. However, latest when it come to compile the swift module causing the issue, the problem will occur. Forcing quit of 'swift-frontend' or 'SourceKitService' before starting a build activity helps doing some housekeeping and getting 'faster' through build process. However, be patient. my 32GB RAM and additional swap gets consumed by a simple swift module modification. Hope this is coming to an end soon.

I do confirm: Version 13.2.1 (13C100) does not resolve my latest issues in complex project. Compile time is extreme and memory consumption is still > 40 GB!

Xcode 13.2 RC fixes the issue in 2 separate projects (iOS/WatchOS). Hope this works for you all!