I think I've found a proper
solution for this. Apple seems to have change code signing from iOS14 -> iOS15.
Automagically created certificates is of the type "iOS development" and for som reason when these are used on iOS15 they need online verification on every launch.
Deleting my developer certificate and creating a new one of the new type Apple development, solved my issue.
The code signing now only has to be done online at the first launch - just like in the good old days.
I'd wish that Apple would stop messing with the code signing. Every time they do, they make these crazy edge case fuck-ups.
For me the proper fix was to delete provisioning profile and development certificate and create a new of the "new" type "Apple development"