Turning off canvas prevents to run in this situation, i.e. background activities do not lither the memory and lead to the experienced misbehaviour.
However, latest when it come to compile the swift module causing the issue, the problem will occur.
Forcing quit of 'swift-frontend' or 'SourceKitService' before starting a build activity helps doing some housekeeping and getting 'faster' through build process. However, be patient. my 32GB RAM and additional swap gets consumed by a simple swift module modification. Hope this is coming to an end soon.