Model w/ Guardrails Disabled Still Frequently Refuses to Summarize Text

Question

Hunter OP

Created 2d

Replies 4

Boosts 0

Participants 2

Foundation Models are driving me up the wall.

My use case: A news app - I want to summarize news articles. Sounds like a perfect use for the added-in-beta-5 "no guardrails" mode for text-to-text transformations...

... and it's true, I don't get guardrails exceptions anymore but now, the model itself frequently refuses to summarize stuff which in a way is even worse as I have to parse the output text to figure out if it failed instead of getting an exception. I mostly worked that out with my system instructions but still, the refusing to summarize makes it really tough to use.

I instructed the model to tell me why it failed if that happens.

Examples of various refusals for news articles from major sources:

"The article mentions "Visual Lookup" but does not provide details about how it integrates with iOS 26."

"The article includes unsafe content regarding a political figure's potential influence over the Federal Reserve board, which is against my guidelines."

"the article contains unsafe content."

"The article is biased and opinionated and focuses on the author's opinion."

(this is despite the instructions specifically asking for a neutral summary - I am asking it to not use bias in the output but it still refuses)

I have tons of these. Note that if I don't use the "no guardrails" mode and use a Generable instead, some of these work fine so right now I have to do two passes on much of the content since I never know which one will work.

Having a "summary mode" that often refuses to summarize current news articles (the world is not a great place, some of these stories are a bummer) is near worthless.

Boost

Answer 1

Hunter OP

2d

I know someone will ask if I have filed Feedback.

I have in the past and I will file a new one. That said, none of my Foundation Models feedbacks have any activity on them and I've filed about a dozen since June. Feels like a waste of my time.

0

Answer 2

DTS Engineer OP

Apple

6h

Hi @Hunter,

If I'm understanding correctly, you're asking Foundation Models to summarize an article, but it is instead responding with the article verbatim. Is this correct? Or is it simply refusing to respond at all? Either way, yes, the proper path forward would be to file Feedback.

When you do, be sure to included a focused code sample, along with an excerpt of the text that was not properly summarized, so that we can recreate and pinpoint the issue. Please also mention the FB number here so I can keep track of it internally.

As for asking the model itself why it did not summarize the articles, this response should not be treated as an accurate error message due to the nature of probabilistic generation.

Best,

-J

0

Answer 3

Hunter OP

4h

Hi,

No, it's not responding verbatim, sorry if unclear. It's responding with a single sentence saying some variation of "I cannot fulfill this request because..." like you see in the above post.

It will give a reason but it sounds like you're saying that's likely hallucinated so perhaps just nonsense. But it's just a refusal, not the original article text returned back to me.

I seem to see this most consistently on political stories but also strangely on tech stories (it will sometimes refuse to summarize any Apple related rumors which I'm sure is not intentional but pretty odd).

I will post my FBs.

0

Answer 4

DTS Engineer OP

Apple

4h

Hi @Hunter, thanks for clarifying. That is indeed strange behavior. If the model is generating what amounts to a refusal, then that's obviously not something you want to display to users of your app.

I'll attempt to recreate. Please keep me posted, and let me know once you have an FB#.

-J

0