Hey everyone, hoping to compare notes on something we have been chasing since beta 5.
We have a Mac app that uses FoundationModels with @Generable types for structured output. Starting with macOS 27 beta 5, guided generation requests began logging tokenizer errors and our longer structured requests slowed from seconds to minutes. We are still seeing the same thing on beta 6 and beta 7. We filed it as FB24310823 on August 11 with a sysdiagnose and log captures.
The signature is easy to check if you want to see whether your machine does it too. Stream the log while your app generates:
log stream --predicate 'subsystem == "com.apple.tokengenerationcore"' --style compact
On our machine the inference service (TGOnDeviceInferenceProviderService, category guided) prints these two lines in matched pairs, thousands of times:
Generated an empty mask at recognizer index N allowedTokenIDs is empty. Something is likely wrong with the tokenizer
Some numbers from beta 7 today: 9,008 of those pairs in about five and a half minutes. The errors start about one second into the first request after a fresh app launch. Requests that normally finish in 4 to 12 seconds take 77 to 170 seconds or longer. On beta 5 we measured decode at roughly 0.3 tokens per second on the worst requests. Short requests still finish at normal speed but they emit the same errors while they run, and the structured content they return looks degraded to us. On betas 5 and 6 we also saw repeated asset release errors for instruct_300m.tokenizer and the instruct_3b tokenizer saying the asset is not marked as in use.
For what it is worth, a build that ran clean on beta 4 shows the same behavior on beta 5 and later with no app changes, and the same @Generable schema drives both the fast and the slow requests. But we know that does not rule out something on our side, and we would honestly be happy to learn this is our own bug since that would mean we can fix it.
So two questions. Is anyone else seeing this since beta 5? And if you spot something we might be doing wrong on our end, sessions we should be recreating, schema patterns that stress the constrained decoder, anything at all, we would really appreciate the feedback. If it does turn out you are hitting the same thing, a Feedback referencing FB24310823 would help a lot.
Thanks!
