After some additional investigation, turning off Manage Version and Build Number during export fixed it for me. If you upload to TestFlight from the command line, you might need to set manageAppVersionAndBuildNumber to false in exportOptions.plist. I expected that this option will only affect the final app executable and not embedded frameworks.

For me, the different name of the bundle didn't work for binary XCFramework. However, turning off Manage Version and Build Number worked for both binary framework and a framework built from sources.