Update: we found what triggers this, and worked around it.
Short version, it's the number of requests you have in flight on a single connection at the same time. We were multiplexing 20 pushes onto one connection, which is the pushok library's default. Dropping that to one request at a time made the problem go away entirely.
I also need to correct something from my earlier post. I made a lot of the split between address ranges, and that turned out to be misleading. When we tested directly against the ranges that looked clean in our traffic, they failed too, as long as we sent 20 requests at once. The reason those ranges looked healthy is that most of our pushes go out in small batches, small enough to get through. So the address ranges were never really the issue. Batch size was, and the prefix pattern was just a side effect of which traffic happened to land where.
Here's the test we ran. We sent 20 pushes with deliberately invalid device tokens, so nothing could actually be delivered, against several different APNs IPs. The only thing we changed between runs was the concurrency:
20 requests multiplexed onto 1 connection: failed 9 out of 9 times
5 requests per connection: worked 7 out of 8 times
1 request per connection: worked 9 out of 9 times
Leaving the connection open between batches made no difference at all, so it isn't about connection reuse or how old the connection is.
The change we shipped was setting concurrent requests to 1 and concurrent connections to 5 on our APNs client library.
Before the deploy we were failing 85-90% of pushes. Within about twenty minutes of it going out we were at zero, and our push volume is back to what it was before any of this started. It costs us roughly half a second per job, which is a trade we're happy with.
If you're on a different HTTP/2 client, look for whatever controls how many requests share a connection.
One last note on the rollout. By late on the 4th, nearly every address range we touch was affected, not just the ones I listed earlier. So if you tested a while back and concluded some servers were fine, that may not hold anymore.
Topic:
App & System Services
SubTopic:
Notifications
Tags: