Had to downgrade tensorflow-macos to version 2.9.0 and tensorflow-metal to 0.5.0 to get rid of such errors. Others on this site have reported similar issues with more recent versions. Apple is yet to provide working versions for newer TF releases.

Still a problem on M2 (with 16GB of unified RAM). And still stuck on tensorflow-macos 2.9 and tensorflow-metal 0.5.0 since newer versions are broken.