So I tried restarting from the backup of my last working version. I recompiled module Swift stuff and this time my main project was able to compile! Yay! (Of course right after I post here). I have no idea what is different about this older project file than a new one.

I of course tried to compile my project (the working version) with the new Xcode but I get a Module compiled with Swift 5.9.2 can't be imported by the Swift 6.1.2 compiler. Unfortunately re-compiling those modules in the new Xcode doesn't seem to help.