Started the FB here with sysdiagnose and basic info, will work on sample. FB23643759

Yes, same results on device - same log messages, same lack of results. A bug report and a sysdiagnose is not a problem. A sample that reproduces I can put on my list though that may take some more time. This is part of a big project, not sure how much I'll have to extract to get it to reproduce. In the meantime, if anyone else sees this I'd love to hear about it. Or if you're using SpotlightSearchTool with great success I'd be interested in that too! FB number incoming...

Fair enough. Testing on a beta 3 device (iPhone 17 Pro Max), I get the identical console logging about the failure.

Looking around, sounds like perhaps the 4096 also needs to hold tokens beyond the content? If correct, it would be nice to have this broken out better in the error or something similar because just reading this makes it sound like a framework bug.

Thanks for the engagement on this and I appreciate you taking a look.

@DTS Engineer Here is an example feedback taken from Playgrounds. If whatever it captures+provides does not include what you are looking for let me know and I can add more to it. The session instructions are trying to adhere to requirements like the acceptable use doc linked. Trying to! FB19916776

Actually, I didn't know about the acceptable use guidelines and I thought I had read all of the various documentation bits on Foundation Models so good to have that link for sure. Been busy the past few days but I will post the sample/FB, yes.

Hi, No, it's not responding verbatim, sorry if unclear. It's responding with a single sentence saying some variation of "I cannot fulfill this request because..." like you see in the above post. It will give a reason but it sounds like you're saying that's likely hallucinated so perhaps just nonsense. But it's just a refusal, not the original article text returned back to me. I seem to see this most consistently on political stories but also strangely on tech stories (it will sometimes refuse to summarize any Apple related rumors which I'm sure is not intentional but pretty odd). I will post my FBs.

I know someone will ask if I have filed Feedback. I have in the past and I will file a new one. That said, none of my Foundation Models feedbacks have any activity on them and I've filed about a dozen since June. Feels like a waste of my time.

Check seed 4! Seems better. I'm still evaluating but promising.

For those watching, seed 4 seems much improved.

Our long national nightmare may be over - testing with seed 4, the guardrails seem back to being sane. It made it into the release notes as a bug which I take as a positive sign that it wasn't a policy thing. Phew.

Sure, here are a couple, including one that is reproduced with WWDC sample code from Apple. FB18787534 FB18712543 Also, in case you have not seen this thread, there is additional discussion: https://developer.apple.com/forums/thread/792022 Question - do you know if adapter training could help with guardrails issues? I was assuming that the guardrails either ran in a separate process/mechanism or were deeply embedded enough in the training process that it wouldn't matter but if I'm wrong about that and training a custom adapter could help, that's something I would definitely consider (I have a lot of good data, perfect for this sort of training).

Hi J, Thanks for the reply. A little more info: In my specific case, I am indeed using guided generation already (which helped a lot in seed 1 and 2). I thought the eval session was great. I've been working with LLMs for a bit now and it's awesome to have a whole WWDC session to talk about evals and expose them to devs who may not have seen them before. This is coming up for me now because something changed in seed 3 where my feature went from a ~95% success rate to a 0% success rate, all failing with guardrails errors that did not trigger in the first two seeds. There are a bunch of other threads here on the topic and I've filed several feedbacks on the specifics already. Maybe that's a bug/unexpected outcome and we'll see a future seed restore the behavior. I hope so, I'd like to ship this feature. But if not, at least I won't have sent it to customers. My real concern is that in a future point release where the amount of feedback time is compressed and IMO it's very hard to get specific issues in front of engineers via the Feedback system with enough time to get a fix done and tested, this will pop up again. Since the models are non-deterministic, unless you're running my evals, there's a good chance the team may not even know about what's for me a serious regression until it ships. I really want to use and love the framework because it's so promising but I'm a little freaked out now given what's happening in the current seeds. That's the background for my concern and I don't think there's any way to mitigate that at the moment (right?)

Having no reasonable assurance that a feature I build against these might not stop working outside of my control does freak me out quite a bit, especially since we're already seeing massive changes in what's allowed in the current seeds. FB18924722