"Assertion failed: (false) function _onqueue_rdar53306264_addWaiter file TubeManager.cpp line 1042" Crash

We are experiencing a large number of crashes in our production environment, mainly occurring on iOS 16 systems and iPhone 8 and iPhone X devices. The crash log and stack trace are as follows:

Error: Assertion failed: (false) function _onqueue_rdar53306264_addWaiter file TubeManager.cpp line 1042

Crashed: com.apple.CFNetwork.LoaderQ

0 libsystem_kernel.dylib 0x7198 __pthread_kill + 8

1 libsystem_pthread.dylib 0xd5f8 pthread_kill + 208

2 libsystem_c.dylib 0x1c4b8 abort + 124

3 libsystem_c.dylib 0x70d8c err + 266

4 CFNetwork 0x1eb80 CFURLRequestSetMainDocumentURL + 6288

5 CFNetwork 0x44fd8 CFURLCacheRemoveAllCachedResponses + 22624

6 CFNetwork 0x39460 _CFHostIsDomainTopLevel + 968

7 CFNetwork 0x1f754 CFURLRequestSetMainDocumentURL + 9316

8 CFNetwork 0x233e0 CFURLRequestSetRequestPriority + 8792

9 CFNetwork 0x20d38 CFURLRequestCopyHTTPRequestBodyStream + 1612

10 CFNetwork 0x4f950 CFHTTPCookieStorageCopyCookies + 16276

11 CFNetwork 0x15878 CFURLRequestSetURL + 7600

12 libdispatch.dylib 0x637a8 _dispatch_call_block_and_release + 24

13 libdispatch.dylib 0x64780 _dispatch_client_callout + 16

14 libdispatch.dylib 0x3f6fc _dispatch_lane_serial_drain$VARIANT$armv81 + 600

15 libdispatch.dylib 0x401e4 _dispatch_lane_invoke$VARIANT$armv81 + 432

16 libdispatch.dylib 0x41304 _dispatch_workloop_invoke$VARIANT$armv81 + 1620

17 libdispatch.dylib 0x49f14 _dispatch_workloop_worker_thread + 608

18 libsystem_pthread.dylib 0x1bd0 _pthread_wqthread + 284

19 libsystem_pthread.dylib 0x1720 start_wqthread + 8

Have you encountered a similar issue before?

Answered by DTS Engineer in 854019022

Thanks for the .ips file.

Using that crash report I was able to track down a bunch of different bug reports about crashes with similar signatures. My reading of the ‘lead’ bug (r. 54802623) is that we believe this is fixed by iOS 17 and later. Have you seen any crashes like this after iOS 16?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

For those reading along at home, zhenYang sent me full copies of their crash reports via a different channel. Which is why I haven’t asked for them here (-:

mainly occurring on iOS 16 systems

Mainly? Or exclusively?

What’s the most modern OS release where you’ve seen this crash?

Also, are you able to get a JSON format crash report (.ips) for this issue? If so, please post it here. That opens up more avenues for me to investigate this internally.

See Posting a Crash Report for advice on how to post crash reports.

ps The Radar bug indicated by that symbol name (r. 53306264) was resolved in iOS 13, so it’s unlikely to be a concern here.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Right. The second file is from a third-party crash reporter, which is rarely helpful [1]. The second is from the Apple crash reporter, but it’s in human readable format (.crash). I’m looking for a crash report in the JSON format (.ips). Do you have one of those?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] For an explanation of why third-party crash reporters are a bad idea, see Implementing Your Own Crash Reporter.

I have found an IPS file and have sent it to you via email. Please check your inbox. Case-ID: 15414267 linkText

This is an ips crash log

Thanks for the .ips file.

Using that crash report I was able to track down a bunch of different bug reports about crashes with similar signatures. My reading of the ‘lead’ bug (r. 54802623) is that we believe this is fixed by iOS 17 and later. Have you seen any crashes like this after iOS 16?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

We observed the crash rdar53306264 in both Xcode → Organizer → Crashes and our third-party crash monitoring system. It has been occurring on both iOS 16 and iOS 17, and we did not find the r.54802623 bug.

Another concern is that this crash appeared in large volumes during a certain period of time. Could you please advise what might have caused this? We are worried that it may happen again in the future.

we did not find the r.54802623 bug.

Right. That was an internal bug and you don’t have access to it. I talk more about why I post such bug numbers in Bug Reporting: How and Why?

Could you please advise what might have caused this?

There’s no easy answer to that question. Historically this crash has been triggered by a range of concurrency problems within CFNetwork. However, as I mentioned previously, we believe that those were fixed by iOS 17. So, if you’re seeing this crash in iOS 17 and later, then this is a new bug with similar symptoms.

Are you seeing this on iOS 18?

That matters because it’s reasonable to file a bug against iOS 18. However, if the problem is limited to iOS 17 then there’s little point filing a bug about this because, even if we were able to find and fix it, there’s no release vehicle for iOS 17 fixes like this.

If you have an .ips crash report for the crash you’re seeing on iOS 17, please post that here and I’ll take another look at this.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

It’s better to reply as a reply, rather than in the comments; see Quinn’s Top Ten DevForums Tips for this and other titbits.

What modifications can we make to prevent this issue from reoccurring?

Honestly, I’ve no idea. Earlier I wrote:

Historically this crash has been triggered by a range of concurrency problems within CFNetwork

The nature of concurrency bugs is that they can come and go based on hard to understand criteria. Consider this thought experiment:

  1. CFNetwork has a concurrency bug that causes this crash. The bug is highly dependent on network timing.
  2. You see a huge uptick of crash reports in your app during late November and late December.

This might be nothing to do with your code, or your infrastructure. Rather, a big chunk of your users are travelling in late November and late December due to the US Thanksgiving and Christmas holidays. While travelling they’re on WWAN rather than Wi-Fi, which disrupts the network timing, which triggers this bug.

Now, I’m not proposing this as an explanation of what you’re seeing. Rather, it’s an example of just how hard it is to draw conclusion from crash reports like this.

I had a look at the various bugs mentioned in this thread and, sadly, there’s nothing concrete I can point you at. There were a range of different state management bugs, all resulting in crashes like this. Some of those bugs were very hard to track down, so we added extra debugging infrastructure to CFNetwork to aid in the investigation. That’s why you see the TubeManager::_onqueue_rdar53306264_addWaiter(…) symbol.

This was a multi-year investigation, and eventually we got to the point where these crash reports stopped arriving in significant numbers. That’s good, but it makes it hard for me to say “This crash is caused by this sequence of events.”

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

"Assertion failed: (false) function _onqueue_rdar53306264_addWaiter file TubeManager.cpp line 1042" Crash
 
 
Q