I noticed the memory increase is mainly from swap memory. I compared keras.fit with different input size of image, the training memory of small size rarely increased but the larger one increased very much. Maybe it is related to macOS swap memory issue?