I found a solution that ended up fixing the freezing! TLDR: set the gesture's coordinateSpace to .global when instantiating.