Hi There, Kindly look into the CPUs, GPUs and RAM usage (and fan speed along with the Temperature of the CPUs/GPUs at the top of the image). Sorry if the image quality is bad! Takeaway: CPU/GPU usage is extremely POOR, perhaps due to memory leakage and sub-optimal process scheduling among CPUs/GPUs. Environment: TF-MACOS==2.9.2 and TF-METAL=0.5.1 along with python 3.9. I AM STUCK. KINDLY HELP. --Bapi

CPU Only run on Mac STUDIO (20c CPU, 64c GPU, 128GB RAM). Training is STALLED, perhaps the CPUs are DEAD for some FABULOUS REASONS. Below is the snapshot (with temperatures of different cores).

HEY, ANY UPDATE? SHOULD MY 64c GPUs BE ALLOWED TO SIT IDLE?

Just to add, I see nothing of these characters while running with TF-MACOS=2.8.0 and TF-METAL=0.4.0 for python=3.8.13 env.

even if you use keras.backend.clear_session(), it does not help. only saviour here is the CPU mode. But this is not why paid hefty money for the GPU cores! just FYI, TF2.10/METAL-0.6 with python 3.10.6 in CPU mode got me segmentation fault. may be it is specific to me!

Still no solution!!! Environment: TF-MACOS==2.9.2 and TF-METAL=0.5.1 along with python 3.9 in CPU mode ONLY. Memory usage had gone up to 162GB (RAM is 128GB). Strange thing is step time has increased from ~140ms to 64s to hopping 496s before being STUCK? How could someone use these BRITTLE (METAL) GPUs? :-( 1/1 [==============================] - 0s 144ms/step 1/1 [==============================] - 0s 141ms/step 1/1 [==============================] - 0s 142ms/step **1/1 [==============================] - 64s 64s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step 1/1 [==============================] - 496s 496s/step**

I don't see any improvement after upgrading to MacOS Ventura. Only difference is the step time has reduced to ~80ms vs ~140ms. Rest remains the same.