I just ran into the same thing. My guess is this is a bug in the CoreML compiler or on-device scheduler, where it tries to put part of the network onto the Neural Engine, even though it contains convolutions that need more memory than the Neural Engine can handle.

This was recently acknowledged as an Xcode 13 bug here: https://github.com/apple/coremltools/issues/1301

Perfect, thank you for the quick reply and excellent workaround! Much appreciated!