Beta 4 finally fixed it for me!

My bug report got flagged as ”Potential fix identified - For a future OS update” and ”More than 10 recent similar reports”. So please remember to file a bug report in the feedback app to make it a higher priority for Apple.