Notarization submissions stuck in “In Progress” for >24 hours (CI / REST API) – production impact

We are experiencing notarization submissions that remain in the “In Progress” state for an extended period (over 24 hours), with no status transition and no submission log available.

This is occurring in an automated CI environment using the Notary REST API (non-interactive submission and polling). Re-submitting the same package only results in additional submissions also stuck in “In Progress”.

There does not appear to be any API mechanism to cancel, clear, or expire these submissions once they are created.

We have already opened an Apple Developer Support case regarding this issue (Case ID: 102818066745 & 102819008943), but have not yet received clarification on what is causing these long-running “In Progress” states.

This issue is impacting our production release pipeline, as we are unable to reliably complete notarization for signed packages within an expected timeframe.

Based on other reports in this forum (including thread 811968), this behavior appears similar to cases where notarization requests were delayed due to backend backlog or in-depth analysis.

We would appreciate clarification on the following:

  1. Is it expected behavior for notarization submissions to remain in “In Progress” for such a long period without logs?
  2. Is client-side timeout and re-submission the recommended handling for CI workflows?
  3. Are there known service-side conditions (e.g. analysis backlog) that could explain this behavior?

Any guidance from Apple DTS or others who have encountered this would be greatly appreciated.

I have a standard spiel about this which I’ve included at the end of this reply. Please read it through first.

With that in mind, let’s come back to your specific questions:

Is it expected behavior for notarization submissions to remain in “In Progress” for such a long period without logs?

Yes, assuming you’ve hit the “in-depth analysis” path.

Is client-side timeout and re-submission the recommended handling for CI workflows?

I see two common patterns reported here on the forums:

  • Developers hit this on their first notarisation attempt, or otherwise very early in their Mac development ‘journey’.
  • Developers have been happily notarising for a while and then suddenly bump into this.

In the first case my advice is to simply wait. That’s the focus of the boilerplate text that I included at the end of this reply.

My understanding is that you fall into the second case. If you were a small shop that was notarising manually, I’d give you the same advice. Usually this state clears in a few days and so it’s easiest to simply wait.

However, handling this in a CI system is tricky because there’s no clear way to identify this state. I generally don’t recommend just timing out and retrying the exact same submission. This is based on this advice from the notary team. Specifically this:

If your app falls into this category, rest assured that we’ve received your file and will complete the analysis, though it may take longer than usual.

and this:

In addition, if you’ve made changes to your app while a prior upload has been delayed, it’s fine to upload a new build.

from Q&A with the Mac notary service team, linked to from Notarisation Resources (my emphasis).

Resubmitting the exact same build just leaves the notary service with more backlog to grind through once the state clears. OTOH, if you don’t retry at all then you can’t really be sure whether you’ve hit the “in-depth analysis” path or something else has gone wrong.

One option is to let your CI system continue to submit requests up to a point. The symptoms of this state are that, once you enter it, your submission and all subsequent submissions stay In Progress until it clears. So, if your CI system submits N requests — perhaps N might be 5 — and they all stay In Progress, then you can reasonably infer that you’re in this state. At that point it might make sense to hold off new submissions until the state clears.

Having said that, this complexity isn’t required. As noted above, it’s fine to continue submitting new builds while in this state, and that’s certainly the easiest option.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"


You can expect that most uploads will be notarised quickly. Occasionally, some uploads are held for in-depth analysis and may take longer to complete. As you notarise your apps, the system will learn how to recognise them, and you should see fewer delays.

For lots of additional info about notarisation, see Notarisation Resources. Specifically, it links to a Q&A with the notary service team that’s quite instructive.

Thanks for the detailed explanation — that’s very helpful.

Based on your description, this does seem to align with the second case you mentioned — a mature notarization workflow that suddenly enters the in-depth analysis path.

To add some context from our side: we’ve been notarizing successfully via the Notary API in an automated CI workflow for quite a long time without issues. This behavior (submissions remaining in “In Progress” for an extended period with no logs) only started appearing within the past week or so. Prior to that, our notarization requests were completing normally and consistently.

Out of curiosity, are you aware of any recent service-side changes or increases in backlog that might explain why established workflows could start hitting this path more frequently?

If possible, could you let us know whether anything looks unexpected from the service perspective, or whether this still appears consistent with the in-depth analysis state? Please also let us know if there’s any additional information we can provide to help with that confirmation.

Out of curiosity, are you aware of any recent service-side changes or increases in backlog … ?

No. Out and out outages are usually signalled via Developer > System Status > Developer ID Notary Service.

In many cases this isn’t update as promptly as I’d like, so the status indicates a fixed issue rather than an active outage. However, my experience is that notary outages are often related to underlying issues in Apple’s infrastructure, so if I suspect an outage I look an that page and the wider Apple System Status for other problems.

In terms of backlog, that’s not usually a problem although there was this one exception.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Notarization submissions stuck in “In Progress” for >24 hours (CI / REST API) – production impact
 
 
Q