Filed FB18712543

Yeah. Guardrails are part of the system but the change in this beta is so severe, they trigger almost all the time. I suggest we all file Feedbacks and quickly. We're already mid-July... I'll post my FB number once I have it all written up.

I've found that the prompts I would use for an API based current SOTA LLM really don't work that well with this little guy. Probably to be expected but after a lot of fine-tuning I've gotten some acceptable results. Also, this is the worst they are ever gonna be. :-)

Since installing beta 3, literally every single request I send to Foundation Models generates a guardrails error where before it was about 10%. Since we don't have Xcode yet, I haven't re-tested my evals but it seems like something really big changed in this seed. I would call it a big regression to the point of no longer being unusable.

It's on Apple's end. You gotta wait for them to fix it.

I think what you did - adding nonisolated - is correct. Also needed for SwiftData and at-ModelActor.

I updated the report with more examples. It's really, really sensitive. Any news article about someone dying for instance, rejected.

In this case, it was a foregrounded visionOS app. This app is a feed reader / RSS app and this happened when I downloaded new feed items (~150 of them) and ran the model in a loop over each entry. The actual model request is to summarize the content of the feed item. FB17984127 The error is: Error Domain=com.apple.SensitiveContentAnalysisML Code=15 "Failed model manager query for model com.apple.fm.language.instruct_300m.safety: Client rate limit exceeded, try again later"

Pretty hacky but thanks for sharing, will try it out. Hopefully this is fixed in the next release. I'll be at WWDC and this is on my list to try to ask about, either in-person or in one of the virtual labs.

It's not ideal but it took me about 10 minutes to revert it so doesn't seem like the end of the world, at least for most implementations. Kevin's signature says he is a DTS engineer, that means Developer Technical Support which at least commonly would suggest he's not the person who would be assigned to work on fixing a bug like this but to help developers understand the APIs and work around issues. I appreciate him giving us such detailed information - that's not always the case.

Interesting - I left Xcode open overnight and when I woke up, the SDK was there.

Yup, I guess that's the next step, a sample. I am submitting the required elements (nicely documented, appreciate that). What I'm noticing is that it almost appears throttled (though the delegate is not getting the throttling message). On Vision Pro, it appears that I'm much more likely to get a response if the battery pack is plugged into mains power... but nobody really uses the Vision Pro that way so that's not something I can rely on. Anyway - thanks for the tip.

Looks like this may now be fixed - can others confirm?

Looks like this may now be fixed.

Anyone have any idea how to escalate this? Seems insane.