Looking around, sounds like perhaps the 4096 also needs to hold tokens beyond the content? If correct, it would be nice to have this broken out better in the error or something similar because just reading this makes it sound like a framework bug.

Thanks for the engagement on this and I appreciate you taking a look.

@DTS Engineer Here is an example feedback taken from Playgrounds. If whatever it captures+provides does not include what you are looking for let me know and I can add more to it. The session instructions are trying to adhere to requirements like the acceptable use doc linked. Trying to! FB19916776

Actually, I didn't know about the acceptable use guidelines and I thought I had read all of the various documentation bits on Foundation Models so good to have that link for sure. Been busy the past few days but I will post the sample/FB, yes.

Hi, No, it's not responding verbatim, sorry if unclear. It's responding with a single sentence saying some variation of "I cannot fulfill this request because..." like you see in the above post. It will give a reason but it sounds like you're saying that's likely hallucinated so perhaps just nonsense. But it's just a refusal, not the original article text returned back to me. I seem to see this most consistently on political stories but also strangely on tech stories (it will sometimes refuse to summarize any Apple related rumors which I'm sure is not intentional but pretty odd). I will post my FBs.

I know someone will ask if I have filed Feedback. I have in the past and I will file a new one. That said, none of my Foundation Models feedbacks have any activity on them and I've filed about a dozen since June. Feels like a waste of my time.

Check seed 4! Seems better. I'm still evaluating but promising.

For those watching, seed 4 seems much improved.

Our long national nightmare may be over - testing with seed 4, the guardrails seem back to being sane. It made it into the release notes as a bug which I take as a positive sign that it wasn't a policy thing. Phew.

Sure, here are a couple, including one that is reproduced with WWDC sample code from Apple. FB18787534 FB18712543 Also, in case you have not seen this thread, there is additional discussion: https://developer.apple.com/forums/thread/792022 Question - do you know if adapter training could help with guardrails issues? I was assuming that the guardrails either ran in a separate process/mechanism or were deeply embedded enough in the training process that it wouldn't matter but if I'm wrong about that and training a custom adapter could help, that's something I would definitely consider (I have a lot of good data, perfect for this sort of training).

Hi J, Thanks for the reply. A little more info: In my specific case, I am indeed using guided generation already (which helped a lot in seed 1 and 2). I thought the eval session was great. I've been working with LLMs for a bit now and it's awesome to have a whole WWDC session to talk about evals and expose them to devs who may not have seen them before. This is coming up for me now because something changed in seed 3 where my feature went from a ~95% success rate to a 0% success rate, all failing with guardrails errors that did not trigger in the first two seeds. There are a bunch of other threads here on the topic and I've filed several feedbacks on the specifics already. Maybe that's a bug/unexpected outcome and we'll see a future seed restore the behavior. I hope so, I'd like to ship this feature. But if not, at least I won't have sent it to customers. My real concern is that in a future point release where the amount of feedback time is compressed and IMO it's very hard to get specific issues in front of engineers via the Feedback system with enough time to get a fix done and tested, this will pop up again. Since the models are non-deterministic, unless you're running my evals, there's a good chance the team may not even know about what's for me a serious regression until it ships. I really want to use and love the framework because it's so promising but I'm a little freaked out now given what's happening in the current seeds. That's the background for my concern and I don't think there's any way to mitigate that at the moment (right?)

Having no reasonable assurance that a feature I build against these might not stop working outside of my control does freak me out quite a bit, especially since we're already seeing massive changes in what's allowed in the current seeds. FB18924722

Yeah, it seems much more restrictive in b3. We've got another thread going with lots of examples also. File feedback! https://developer.apple.com/forums/thread/792022?page=2

Not in this release... fingers crossed for some time in the future.

I filed a second one with my own content, FB18787534. Just to share a public example, this is an example of text the model won't summarize saying it's "unsafe". Seems pretty anodyne to me... let alone trying to summarize news about current events, obits, etc. Here’s a product recommendation long in the making. Four years ago this month, Matthew Panzarino was my guest on The Talk Show and at one point he recommended Moft’s Snap-On iPhone Stand/Wallet. It uses a very clever origami-style folding design. Folded flat it kind of just looks like a leather MagSafe wallet. But folded open it works as a stand — and as a stand, it works both horizontally and vertically. Borrowing images from Moft’s website:You can also use it with the stand oriented vertically but the phone horizontally.I bought one of these right after that episode of the show, and I’ve been using it ever since. And every so often when I use it, I think to myself that I should write a post recommending it. I’ve waited so long that Panzarino has been back on The Talk Show five times since the episode in which he recommended it, but here we are. The thing is, I use it both in my kitchen and while travelling, and so I’ll often find myself in the kitchen, rooting around the drawer in which I keep it stashed, only to realize it’s downstairs in my office in my laptop bag. Or, worse, I’ll find myself looking for it in my laptop bag while I’m sitting on an airplane 35,000 feet in the air, only to realize it’s back home in my kitchen. So I ordered a second one today — which I should have done like 3.5 years ago.I own a few similar/competing products, like these PopSocket-y rings from Anker and Belkin. I have no idea why I own both of those rings when I don’t like either of them as much as the Moft foldable stand. The problem with these rings is that they’re only able to prop the phone horizontally. Watching video is almost certainly the most common use case for these stands, but I do often use my iPhone propped up vertically, like for FaceTime calls and when I’m writing on it using a Bluetooth keyboard. I’m going to give both of these rings away — there’s nothing they do better than the Moft stand. The Moft stand even works better as a hand-holding grip.I’ve never used the Moft stand as a wallet, but if you want to, it holds two cards. Prime “Day” lasts a week and it’s still running until midnight Pacific tonight, but the Moft stand doesn’t have a Prime Day discount: it’s the same price at Amazon as it is from Moft’s website: $30. Well worth it. I love this thing. (Buy yours wherever you want, of course, but the Amazon link a few sentences back will throw some filthy affiliate lucre my way.) guardrailViolation(FoundationModels.LanguageModelSession.GenerationError.Context(debugDescription: "May contain sensitive or unsafe content", underlyingErrors: [FoundationModels.LanguageModelSession.GenerationError.guardrailViolation(FoundationModels.LanguageModelSession.GenerationError.Context(debugDescription: "May contain unsafe content", underlyingErrors: []))]))