If you are by any chance using "Turbo Boost Switcher", this is what caused the lag for me.
To solve it I did this:
Quit "Turbo Boost Switcher" (not just disable Turbo Boost)
Go to System Preferences > Displays and change the scaling
Change it back to how you want the scaling to be
The animations were not lagging anymore for me, I was able to replicate this 2 times.
You can even start "Turbo Boost Switcher" again after going through those steps.
But I don't know what actually causes the lags to start, because sometimes they come back.
Topic:
App & System Services
SubTopic:
Core OS
Tags: