I tried to delete the existing session and reload the saved model everytime before prediction. that does not help at all. memory keeps increasing.

Yep. Mac M1/M2 GPU cannot be used for machine learning if this bug is not fixed.