I have the same problem. Sometimes frame capture works correctly, but doesn't work for most of the time and pops up with the same error.

I found that using precise::fract can solve this problem. This is good since I can enable fast-math for other parts of my program. Wish apple can provide more fine-grain fast-math control, e.g. function level, statment block level...