Check out my similar issue - there seems to be some specific sort of operation/layer that doesn't work well: https://developer.apple.com/forums/thread/696870?page=1#699024022

@gtsoukas Yeah this is quite frustrating - I spent a lot of time trying to debug what was wrong with my network only to find out that I hadn't done anything wrong.

@gtsoukas I've found some more weirdness with tf-metal: If I use an unofficial implementation of mobilenet, the validation accuracy is good, but not with the official Tensorflow implementation. Comparing these two should give an indication of what the issue is in my case.

Hey I'm still facing this problem.

That's just silly 🤦‍♂️