The problem was that the focus could only move vertically or horizontally. And because of my .frame() modifier, the buttons were positioned diagonally. The solution was to use .focusSection(). Here it's explained: https://developerinsider.co/swiftui-focussection-uifocusguide-alternative-for-swiftui/

Actually, .navigationViewStyle(.stack) is the answer. I don't know why, it's just better than the usual one.