After wasting hours with this weird behavior, I've accepted that this is a perfect speciment of the "you're holding it wrong"-situation.
Let me put it this way (thinking about this a few times solved my problem):
Whatever you want to render above/below the List (which is why you need the scrollview) can also be put into a separate Segment of the list. It won't look as bad as you might initially think!
Topic:
UI Frameworks
SubTopic:
SwiftUI
Tags: