Still reproduces on AFM 3 Core Advanced. Same repro tool as before, default sampling, no token cap, 15 guided requests across two runs today. 6 of 15 ran until they filled the 8192 token context and threw LanguageModelError, each one taking 159 to 167 seconds to get there (one took 245). The other 9 completed in 14 to 25 seconds with normal output. Beta 7 was 2 of 5 and the 27.0 RC was 4 of 5 on the same machine, so today's number is within the variance we have been seeing, not a change. Same split by content as before. The sections whose prompts produce the longest answers ran away most often (Security 3 of 3, Data safety 2 of 3). The two shortest never did. What is new in this build is the logging. The "allowedTokenIDs is empty" and "Generated an empty mask" lines are gone. TGOnDeviceInferenceProviderService now logs "Token N does not match grammar at recognizer index N" on the same subsystem and category instead. I saw about 106,000 of those in 15 minutes. They appear within a second of every request starting, healthy ones included, so as before the log volume does not tell you which request is about to run away. The recognizer index does: healthy requests stopped between 11k and 16k, runaways passed 45k. Variant and context size are unchanged (AFM 3 Core Advanced, 8192). I have not put this beta on the Core machine yet, so the round 6 control still stands: same requests, 4 to 14 seconds, no runaway on Core. Appended the full log and per request numbers to FB24310823.

Update, September 10, macOS 27.0 (26A428), two machines. We ran the same Thorough scan of the same Android project with the same TestFlight build on a MacBook Pro (M4 Pro) and on a MacBook Neo (A18 Pro, 8 GB). Both on 26A428. We read SystemLanguageModel.default.variant at runtime on each: the MacBook Pro reports AFM 3 Core Advanced with a context size of 8192, the MacBook Neo reports AFM 3 Core with a context size of 4096. AFM 3 Core: five guided requests, 5 to 14 seconds each, whole scan 40 seconds, generation stops normally, zero reviewer considerations. The empty mask and allowedTokenIDs errors still scroll in the log, so the log signature is on both models. AFM 3 Core Advanced: five of ten requests generated until the context window filled and threw the context size error at 159 to 207 seconds each. Whole scan about 15 minutes. The retried sections produced considerations with fabricated evidence. Generation speed is about the same on both. The difference is that Core stops and Core Advanced does not. Every earlier capture on our Feedback was on the M4 Pro, so this is the first clean run we have seen on macOS 27 since beta 4, and it is the Core model. Two questions for anyone who can answer. Does this reproduce for others only on machines that get AFM 3 Core Advanced (M3 or later with 12 GB or more)? And is there any supported way for an app to ask for AFM 3 Core on a machine that has Core Advanced, until the fix lands? The variant property is read only as far as we can see. Details and both captures are on FB24310823.

I also wanted to add that I was testing this on a MacBook Neo using the AFM 3 Core model. I am still seeing that token error in the terminal logs, but of note the MacBook Neo runs the app and more detailed scan faster, cleaner than a MacBook Pro using the AFM 3 Advanced model. The Neo finishes before the Pro and does not add hallucinations as the MBP does.

Update on the macOS 27.0 release candidate, build 26A428, Xcode 27 RC. We reran our fixed set of scans with the log stream capture. Nothing changed from beta 8. Every guided generation request still logs the tokenizer pairs (about 10 to 20 per second, same emitter and category), short requests still finish in 3 to 12 seconds, and long requests still crawl to about 160 seconds with the recognizer index above 50,000 before stopping, which we read as the context window filling. Three of five sections on our busiest project did that tonight, at 159.6, 160.2 and 157.8 seconds. The scan took 10 minutes 43 seconds; a healthy build does it in under a minute. Two things worth adding to the record. First, on the same build the same Google Play instructions ran five sections in under ten seconds each on a project with little to say, so the prompt and the store are not the variable. Output length is. Second, the "Respond in well formatted JSON." line has been in our instructions since August 28 and made no measurable difference on the RC either. I have updated the FB on this and added all the terminal outputs and details.

Would the main suggestion be to break down the Generable even more to see if that could help level things out a bit?

I tested on beta 8 yesterday, updated some details in the FB on this issue. Of note, we are breaking out the Generable into 5 (at least) currently. Think of it as each section of the App Review Guidelines. I have updated a breakdown in the FB on this and some more on the app and what it is doing under the hood. Of note the on-device models are not doing a review so to speak but are being used as a judgement pass and also to generate things like developer discussion points.

Alright, I was thinking about this over the weekend for sure. Breaking it up to see if that helps at all. I also did not add the JSON to the prompt and was going to test that as well. I will let you know once I update the FB with anything else I find. Mostly wanted to give you all some more deep context as to what I am doing here and how I am doing it to make sure you had the full picture!

So tested this over the weekend, added the respond with well formatted JSON to the instructions (I have not added this to the prompt but can test that today if needed) and I am seeing the same results but might have found some addition details. I have updated the feedback with a lot of details on the project we are building and what we found this weekend. Thanks Kevin

You guys rock. I will update my two apps and report back. thank you very much :)

Just wanted to follow up and make sure you had everything you needed from me at this. time? thanks kevin

Alright, sorry for the delay response here. Some things going sideways at work. Anyway, updated the FB with the requested items. Please let me know if you need anything else...Thanks :)

I have updated Feedback FB24310823 with the requested details and sample code as well. Please let me know if you need anything more from my end. Thanks Kevin Keheley

Yeah, I think it did dupe the post and that was on my end, sorry about that. Thank you for the support and please let me know if you need anything else : )

No we are on beta 7 now. We starting building apps for the new AFM 3 early in the beta cycle, we only started seeing this issue when beta 5 came along. In other words, we did not see this in beta 4, 3, 2, 1. We are seeing it in beta 5, 6, 7, so far. I included the Feedback in case anyone else is seeing this issue they can tag that to group them together. Resolution is "Open" with less than 10 other Similar Reports.