When comparing Tensorflow CPU and GPU performance, take special note of the impact of batch size when doing the model.fit(). E.g, here you can find a performance graph related to this: https://github.com/moritzhambach/CPU-vs-GPU-benchmark-on-MNIST