Update: If I exclude the arm64 from the iOS simulator, then the described issue appears. If the exclusion is not there then it works nicely.