The errno values usually show up in a sysdiagnose, if you take a look at the logarchive file around the time the issue took place. See the section under sysdiagnose here. @meaton, thank you very much for that input. That certainly helped in understanding one UDP related test failure on my local setup. The logarchive had a couple of lines of log messages which provided the necessary hint that there was a system level content filter extension which was getting involved in this flow and that was causing the unexpected odd failure. Dealing with the system level content filter extension to not interfere with this test, helped the test pass. Of course, there are several other tests that fail intermittently (and in some cases there's no content filter involved). So I'll will be reviewing those failures through these sysdiagnose data. Hopefully, I'll find the necessary data to narrow them down too. Thanks again.

Thank you for responding @meaton. What are the log messages that are written out during this time? Also are there any errno logs generated here? What I meant by the log message being written out was that the source code (the one I'm guessing is the right one https://github.com/apple-opensource/xnu/blob/master/bsd/netinet/udp_usrreq.c#L1463?) has log statements like the KERNEL_DEBUG. However, I haven't found any kind of log messages in the system logs, that relate to UDP communication when these packets go missing. So I was wondering if there was some way to get any kind of logging generated at runtime on these systems? You note above errno logs in your reply. Is there some specific file/location where those get logged, so that I can take a look? Not every single run shows up this issue, but it does happen consistently enough that I think understanding the real reason for them being dropped would help ruling out any unexpected bugs either in the application or some other place.

Having said that, I can mark a post as Apple Recommended, which is what I’ve done for your thread. Thank you Quinn. That's good enough. I just checked that thread and the Apple Recommended answer appears before the accepted answer, so that's a good thing. Thank you for your help.

macOS 12.4 GA has been released a few days back and even 12.5 Beta version. Both these versions continue to have this bug. On the official feedback issue that was submitted, I haven't received any response. It's a bit disappointing that the Beta OS was tested, an issue was identified and feedback was logged but yet the bug made it to the GA version without receiving any response on the feedback issue. As far as I know, the feedback assistance tool is the official recommended tool for reporting these issues. Is there anything we could have done differently to try and get attention to this issue? Some other channel or issue tracker? The feedback issue has the necessary context and details of which project this affects and how widely that project gets used and the impact of this bug. So we were hoping we would get some attention on it. Furthermore, this isn't really an one off issue, there's at least one other issue which we have filed that hasn't received any response.

Finally, is there a doc/schema which explains the structure/contents of .tbd files. Not that I’m aware of. I agree there there should be and I’d appreciate you filing a bug requesting that. Please post your bug number, just for the record. I have filed FB10013233 requesting documentation for these files.

# not good. sa = util.find_library('System') sb = framework_find('System') sc = framework_find('System.framework') ga = util.find_library('OpenGL') gb = framework_find('OpenGL') gc = framework_find('OpenGL.framework') I'm not an expert in this area and new to this, but are you sure these frameworks are expected to be there in the cache at all? For example, when I extract the dyld cache and look into its contents I don't see any of these frameworks. Neither System.framework nor OpenGL.framework. In fact I don't even see Python.framework in that cache. So your "good" case - is it likely that when Python was installed that it actually installed this framework on the filesystem and your calls to util.find_library('Python') and framework_find('Python') are actually finding it on the filesystem and not from the cache?

On a different thread, Quinn pointed me to the right tool to get this information. Details are available in the reply here - https://developer.apple.com/forums/thread/655588?answerId=712400022#712400022. In short, using dyld_info -dependents <library> will list the dependencies.

Oh, thanks for that context. The good news here is that we do have a way to achieve that goal, namely dyld_info. For example ... Excellent. Thank you very much. This is exactly what I was after. The only reason I used otool was because of my lack of knowledge about dynamic libraries and tools in macOS. If you find yourself in a situation where you need to look inside the dynamic linker shared cache for some reason, and that’s not covered by an existing tool, I encourage to file an enhancement request for better tooling. Make sure to describe your requirements rather than focus on one specific tool. Will certainly do going forward. Finally, is there a doc/schema which explains the structure/contents of .tbd files. Not that I’m aware of. I agree there there should be and I’d appreciate you filing a bug requesting that. Please post your bug number, just for the record. I will do that shortly and post the id here. Slightly off-topic, but just wanted to share some feedback about the "Feedback assistance program" (the one for filing bugs) itself - I find that, that tool/process feels like "/dev/null" where you file an issue and see no updates to it at all. I don't just mean comments/responses, but even basic triaging updates like some human responding saying that it's being looked into. This observation is based on more than one issue that I have filed. The other thing with that tool is, you can't see other open issues - unlike other issue trackers where this is a common feature. So it just feels like an area where the person who has filed the issue is just talking to themselves. I'm really glad that at least the forums here are much more responsive with knowledgeable users. For example, I see that each of these file contents end with ... which at first looked odd to me Indeed. That’s a standard part of YAML. And yeah, I only know about that because I looked it up a few minutes ago (-: Thank you for that detail. I didn't even know it was a YAML format file :)

Hello Quinn, I'm reviving this slightly old thread. Your explanation of the dynamic library cache and its history in this thread has been very useful. I have a few related questions. You note: So in recent Xcode releases we’ve replaced the libraries in your SDKs with stub libraries [2]. That is, there’s no longer a libpcap.dylib library in the SDK, but rather a much smaller libpcap.tbd stub library [3]. It’s a text file (.tbd stands for text based description) so feel free to open it up and take a look. Consider one such .tbd file that's shipped in macOS M1 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/AppKit.framework/AppKit.tbd. This file doesn't contain the dynamic libraries that this library depends on. Is that intentionally left out? Tools like otool would be able to use that info to list the dependencies of the library if that info was present, isn't it? You noted: Given the above reality, the libraries in /usr/lib are no longer needed: ... This is fine if you’re using Xcode, or Apple’s command-line tools, because they know how to look inside an SDK for headers and stub libraries. Is otool considered one of Apple's command line tools? The output of man otool does seem to suggest that it is one. If so, then do you think enhancing otool to seamlessly (or via a new option) look into the dynamic library cache if a path specified to that tool doesn't exist on the filesystem would be a valid enhancement? I ask these questions in context of https://developer.apple.com/forums/thread/705281 where I ended up having to use external tools to extract the cache to get hold of the information I was after. Finally, is there a doc/schema which explains the structure/contents of .tbd files. For example, I see that each of these file contents end with ... which at first looked odd to me and made me check if the rest of the contents in that file was missing.

I didn't find a way to get this working with otool. Nor could I find a tool which would seamlessly work against dependency paths of the system libraries which don't represent a file system path but instead are in within the dynamic library cache. However, I did find a workaround. I found this useful tool https://github.com/keith/dyld-shared-cache-extractor which extracts the dynamic cache library to a directory. The dynamic library cache is available on the filesystem as a file under /System/Library/dyld/ directory. For example, /System/Library/dyld/dyld_shared_cache_arm64e. So you can pass that file as an argument to the tool to extract the contents. Once you extract the cache, you can then use otool to list the dependencies of the system library of your choice, as usual. Do note though that the extracted content is around 2.7 GB.

Thank you very much for replying. I had a quick look at your bug and it’s definitely landed in the right place. I will wait to hear back there, then. One small related question - are changes like these (at this low level of the OS) documented some place in the releases? This is more for future reference so that we can take a look at them before filing any issues.

I found some snippet in an unrelated forum/question where someone had shown how to find the compiler flags at runtime that were passed to zlib. This is a zlib specific API and looks something like: zlibCompileFlags(); The API is here https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da3b647f07d718623f/zlib.h#L1174. This can then be combined with flags of your choice (the flags are noted in the doc of that API). So in my case I added something like: printf("compiler flags %lu\n", zlibCompileFlags()); if ((zlibCompileFlags() & (1 << 9))) { printf("Compiled with ASMV or ASMINF, uses ASM code\n"); } else { printf("No ASM code\n"); } if ((zlibCompileFlags() & (1 << 8))) { printf("Compiled with ZLIB_DEBUG\n"); } else { printf("No ZLIB_DEBUG\n"); } if ((zlibCompileFlags() & (1 << 10))) { printf("Compiled with ZLIB_WINAPI\n"); } else { printf("No ZLIB_WINAPI\n"); } to verify the flags of my interest. The output of that code against MacOS 12.2.1 shows that the ZLIB_DEBUG wasn't enabled while building this library. So essentially, I won't be seeing any logging output from this library at runtime. Having said that, this is a very "intrusive" way since I had to write some C code and link it against this zlib library to invoke this API and get the output. It would help if there was a way to just know what set of commands and options were used to build this zlib library that's shipped in MacOS. Anyone has any ideas?