[Possible solution] We've just tried SWIFT_USE_INTEGRATED_DRIVER=NO in all our targets and it massively brings down incremental compile times!!! Tried with Xcode 14.1 and Mac M1 Pro

Same thing! if you add a new line of code the whole project is recompiled. Always the same time (Xcode 14.1)