@Frameworks Engineer how can we further optimize the memory usage for CoreML? I find that for my model, also around the size of 100+MB, on CPU it takes up ~1GB memory, but on GPU it takes up more than 1.7GB memory. Could we understand further on how memory allocation happens on CPU / GPU / ANE, and if there is a way that we can tune it? (e.g. on GPU, I understand that CoreML uses MPSGraph, so is there a way we can reduce the concurrent ops passed into the MTLCommandQueue to reduce peak memory usage?)

Hi @FrankSchlegel , I am facing the same issue whereby memory allocation with .cpuAndGpu is much higher than .cpuOnly. Also have the same observation that memory is not properly deallocated when running on GPU, even with some autoreleasepool tricks. Do you have any updates on this?