Thanks Matt for the suggestion. Didn't see any ECONNREFUSED or ENOPROTOOPT as searched from the console log. Will double confirm anyway. Regards Richard

Double confirmed that didn't find the ECONNREFUSED or ENOPROTOOPT you mentioned. Some other test cases done: Use only dnsproxy and pipe through all dns requests Same failure after a while Use NWConnection rather than NWUDPSession for appproxy udp pipe though. Same failure after some time. Do without redirect QUIC packets inside appproxy Same failure after some time. Is it a Monterey bug since nothing wrong of this kind found on BigSur?

Fired a TSI: Follow-up: 785557535

bug report as below: It is FB9751370 - macOS. https://feedbackassistant.apple.com/feedback/9751370

Thanks for the feedback. Very interesting. I am trying to trace it down in this regard. Also, I have uploaded in the bug report my appproxyprovider.swift, udp process part. Thanks in advance to take a review. Regards Richard

Hi there, Not sure whether you managed to reproduce the said issue with the shrink down project I shared from bug report channel. Just to update you some new findings from my end: With the same test case: doing dns request with my dns.sh script with only udp pipethrough, but based on Big Sur 11.5.2, I can see the same memory leak: 10 hours results in >300MB memory consumption by the system extension although it begins only with around 20MB. I cannot trace further back as Big Sur 11.5.2 is the last Big Sur installer available from web. I wonder anything ongoing in this regard. Processes: 461 total, 2 running, 459 sleeping, 1737 threads                                   08:20:48 Load Avg: 2.54, 2.06, 1.88 CPU usage: 10.97% user, 5.90% sys, 83.11% idle  SharedLibs: 464M resident, 81M data, 66M linkedit. MemRegions: 64289 total, 4344M resident, 227M private, 2069M shared. PhysMem: 16G used (3092M wired), 47M unused. VM: 2487G vsize, 2321M framework vsize, 0(0) swapins, 0(0) swapouts. Networks: packets: 9547216/12G in, 6532190/610M out. Disks: 2223114/65G read, 2284134/46G written. PID  COMMAND   %CPU TIME   #TH #WQ #POR MEM  PURG CMPR PGRP PPID STATE  BOOSTS  %CPU_ME %CPU_OTHRS UID FAULTS COW   93043 com.familyzo 0.4 14:39.27 21  6  73  318M+ 0B  0B  93043 1  sleeping *0[1]   0.01479 0.00000  0  94013+ 1074

Hi there, Regarding the thread https://developer.apple.com/forums/thread/62310 for profiling system extension, I guess it is the content as quote below. There are some questions as I tried: I have the Info.plist file as below not sure whether it is right. <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> ... <key>MallocStackLogging</key> <string>1</string> <key>MallocStackLoggingNoCompact</key> <string>1</string> </dict> </plist> When I start xcode to attach to the system extension process, it says the process is running as root. So I use sudo /Applications/Xcode.app/Contents/MacOS/Xcode and managed to attach the system extension process, but then it seems not functional properly: dns request cannot pass through anymore (as I pass through all udp process). Anything wrong with that? I tried also sudo open /Applications/Xcode.app but again it cannot attach to system extension process. In case with all that profiling successful, I am not sure whether it can tell where the memory leak happens. Thanks in advance for any suggestion. Regards Richard Thanks! I’ve recently been helping a developer with this in the context of a DTS tech support incident, and I came up with a technique that, while not a full workaround, can help a lot. Here’s the highlights: You can set environment variables in your provider by modifying its Info.plist . For example, adding an entry like this: XPCService EnvironmentVariables FOO bar will set the FOO environment variable to bar . That opens up the possibility of using memory management features enabled by environment variables, for example, those documented in the malloc man page. Of those, MallocStackLoggingNoCompact is the heavy hitter. Once you have MallocStackLoggingNoCompact set, Xcode’s memory graph feature becomes super useful. To wit: Start your provider in the normal way. Attach to it with Xcode (Debug > Attach to Process). In the debug pane, click the memory graph button. You can explore the memory graph interactively with Xcode, but you can also export it to disk (File > Export Memory Graph). The resulting .memgraph file can be passed to a variety of command-line tools for memory analysis, including heap , leaks , and malloc_history (all of which have their own man page). You can learn more about memory graphs in WWDC 2018 Session 416 iOS Memory Deep Dive. Share and Enjoy — Quinn “The Eskimo!” Apple Developer Relations, Developer Technical Support, Core OS/Hardware let myEmail = "eskimo" + "1" + "@apple.com"

By the way, is there any feedback on the shrink down project I shared? Any luck to reproduce the issue? And anything found from the diagnostics package I shared? Thanks and regards Richard

Thanks Matt for the confirmation. Tested with the latest Monterey Beta 12.1 and still see the two issues. Memory leak when doing only piping through. The leak can be >100MB for over night by only piping through udp traffic. This is resolvable by killing the system extension process. 1pkill -9 $(pgrep com.familyzone.macappproxy.fzmacappproxy) Network stack crash after 2~3 hours with below error problem from kernel. 1[C24189 IPv4#334f7330:53 failed socket-flow (satisfied (Path is satisfied), interface: en0, ipv4, dns)] event: flow:failed_connect @0.079s, error Can't assign requested address 2nslookup: isc_socket_bind: address not available This is not recoverable even by uninstalling the system extension. Can you help double confirm you got both reproduced with the code I shared? We have at the moment some customers asking whether they could use Monterey with our software as there are some releases already available. But based on current test result, it is still not safe to use. Thanks in advance for any suggestion. Regards Richard

Yes, it is under terminal command with sudo. The process definitely is running as the filtering is functional and the logs are there. Installed with the sameUnder my Intel and M1 macbookpro: The case of Intel macbookpro BigSur 11.6.1: sudo ps -A|grep macappproxy 23656 ??     0:00.17 /Applications/macappproxy.app/Contents/MacOS/macappproxy 23708 ??     0:01.55 /Library/SystemExtensions/6E63B2AD-405A-424B-BF1C-792A81F25B98/com.familyzone.macappproxy.fzmacappproxy.systemextension/Contents/MacOS/com.familyzone.macappproxy.fzmacappproxy 23795 ttys002  0:00.01 grep macappproxy The case if M1 macbookpro BigSur 11.6.1: sudo ps -A|grep macappproxy 91611 ??     0:00.17 /Applications/macappproxy.app/Contents/MacOS/macappproxy 97980 ttys002  0:00.01 grep macappproxy We can see that under M1, the system extension process is not listed.

Tested with the latest Monterey beta 12.2, same issue is still there: after a couple of hours dns request piping through appproxy, no more network anymore. nslookup: isc_socket_bind: address not available ... [C278 IPv4#4aea03a5:53 failed channel-flow (satisfied (Path is satisfied), interface: en0, ipv4, dns)] event: flow:failed_connect @0.004s, error Can't assign requested address

Hi there, As confirmed with the latest Monterey Beta 12.2 with build number 21D5025f, this issue remains the same. And I believe I have provided the sysdiagnose as well as the minimum project to reproduce the issue (network crash + memory leak with dns request loop). Now that it becomes critical for us since more and more customers are asking when can they use our product based on Monterey. We do need your assistance to speed up the investigation and a clear roadmap for a solution. Thanks in advance for any suggestion. Richard

Thanks a lot for this suggestion: by using netstat -vnp udp | wc -l, it is obvious the difference between BigSur and Monterey. On Monterey, the return value keeps increasing when doing dns request loop from < 100 all the way to thousands; while on BigSur, it is steady at around 46~48! That was tested with the same set of code (same installer of our software) Obviously, there should be something wrong in terms of udp socket close missing with Monterey.

This could be done with a copier state update handler that is propagating state for both sides of the connection (local flow and remote side of connection). Currently it is done like that: any EOF read back, will close both sides flow and connection (session). Does it matter to simplify like that? For your update handler to propagate state, do you have any example code for a reference? Thanks in advance for any suggestion.

Thanks Matt for the reply. Are you noticing any failures for your Network System Extensions when you see these logs? These logs happens periodically within one minute interval. But it seems Network System Extensions are all healthy. Are you seeing these logs for specific flows that your provider is handling? For example system flows? This is a good point. There are some system flows from our system service. Need to verify is it the same without that. Thanks for the suggestion.