Upgrading to macOS 11.6 helped somewhat - after a brief evaluation, debugging seems doable now with "only" 15-30 second pauses. Better than the several minutes observed with 11.3 but still a horrible developer experience (small project size with a few dependencies).