I finally found what the problem was. Just had to start my stateIn multi array with full of zeros and restart the state after not needing it. After this no more label probability came with nan.

Forgot to mention, the core ml model i'm using is a activity classifier