APNs proper behavior when connections are reset without warning

The company I work for sends large amount of traffic to APNs on behalf of our customers, with rates up to 50k/second at peek times.

While a vast majority of the messages are successful or return a proper error code that we can parse and honor, roughly 1.5 million times per day our connections are being closed without warning.

The networking library we use, https://github.com/hyperium/h2, is exposing these as ErrorKind::ConnectionReset or ErrorKind::BrokenPipe.

From our reading of the APNs documentation, it is unclear what the correct response is when this occurs. There is no idempotency key included with our messages, so attempting to resend the message again seems frought with the potential for double sends. However, it some cases it seems clear that at least some of the messages in flight over that connection where never sent out at all.

We found this open source project unconditionally retrying messages in this case.

https://github.com/rpush/rpush/pull/734

Could someone from the APNs team please lay out the expected behavior a client should have when our connection is unexpected closed in this manner?

Thank you for your post. I am not familiar with the third-party library you are using and therefore cannot provide specific feedback on it. I recommend verifying that the library does not swizzle the token, as this must be avoided. Additionally, please confirm that push notifications return an HTTP 200 status; an HTTP 401 response indicates that the device should be unregistered from both your server and the library.

Providing sysdiagnose logs from affected devices would assist in diagnosing the issue, as no error details have been shared. The bundle ID would also be useful for reviewing the logs on our end.

If you want to keep you bundle id private I'd suggest you to submit a code-level support request
so we can discuss this further privately.
When you create the request, indicate that you were referred by me at
Apple and make sure to include a link to this thread.

Albert  WWDR

Thanks for getting back to me.

Yes, I do not expect anyone here to understand the specifics of the networking library we are using in our back end rust service, and it isn't necessary.

The two important parts to note are:

  • This is only a very small percentage (well under 1%) of our requests failing due to connections closure. Within a set of requests (with the same payload to different devices) we will see only a portion of them hit connection resets. If we were incorrectly formatting tokens in the requests, I would expect all them to consistently fail or succeed.
  • In these cases our connection is aburbtly closed while requests are in flight. We do not receive an HTTP status code here to parse, as the connection was broken before it completed normally.

This issue involves the communication from our back end servers to APNs and not APNs to Apple devices in question, so I don't think I can get you any relevant sysdiagnose logs.

APNs not provide an idempotency key to include in our request, so we must understand the exact semantics of when resending requests are safe, otherwise we will send duplicates notifications.

The two things we are trying to learn are:

  • Is this low rate of connection resets expected, or is it due to some issue with our requests that we can fix?
  • Is it safe to retry here without concern of double sending push notifications?

Neither are discussed in the public documentation for connecting to APNs.