Two field incidents show an _nstat_rwlock/UDP socket mutex ABBA deadlock followed by a configd watchdog panic. Raw wait-owner records and UUID-matched KDK stacks identify the same cycle in each incident's earlier configd diagnostic and later panic. Affected systems Both kernels: xnu-13432.1.9~1. Kernel UUIDs are in the attachments. Report AReport BmacOS27.0 (26A428)27.0.1 (26A434)HardwareMac16,10 / T8132Mac15,6 / T6030CloudShell PID87443906Blocked configd threadPID 341, TID 19794593PID 113, TID 309370Our NEDNSProxyProvider can capture CloudShell's DNS UDP traffic and forward it via a separate upstream socket. Provider flows and CloudShell sockets have separate lifetimes. Symptoms and reproduction During DNS UDP forwarding and flow/socket cleanup, CloudShell blocks in final socket close. configd's IPConfigurationAgentQueue blocks creating a UDP socket: a diagnostic is captured after 60 seconds without check-ins, followed by panic/restart after 180 seconds: userspace watchdog timeout: no successful checkins from configd (1 induced crashes) in 180 secondsExpected: socket close completes or returns an error. We have no deterministic reproducer, measured failure rate, or reconstructed per-flow close history. Direct lock-owner evidence Raw waitInfo records (mutex tokens are opaque): Report A, 26A428: thread 18691491: kernel mutex 0x61c831d7f38edbc3 owned by thread 19491850 thread 19491850: krwlock 0xfffffe000cb41730 for writing owned by thread 18691491 thread 19794593: krwlock 0xfffffe000cb41730 for writing owned by thread 18691491Report B, 26A434: thread 36984: kernel mutex 0x44aa649ef3095ae1 owned by thread 12555 thread 12555: krwlock 0xfffffe000ca01730 for writing owned by thread 36984 thread 309370: krwlock 0xfffffe000ca01730 for writing owned by thread 36984In each incident, the first two records form a cycle in both snapshots; the third is configd. All three threads are TH_WAIT / TH_UNINT. UUID-matched kernel stacks Caller-to-callee chains; listed offsets match in both kernels. Kernel worker (A: TID 18691491; B: TID 36984): nstat_idle_check + 808 -> in_pcb_checkstate + 128 -> udp_lock + 200 -> lck_mtx_lock_contended + 1212Holds _nstat_rwlock exclusively; waits for CloudShell's UDP socket mutex. CloudShell (A: TID 19491850; B: TID 12555): soclose + 212 -> soclose_locked + 2300 -> sofreelastref + 60 -> flow_divert_detach + 332 -> in_pcbdetach + 168 -> nstat_pcb_detach + 92 -> lck_rw_lock_exclusive + 180 -> lck_rw_lock_exclusive_gen + 912Holds the socket mutex; waits for _nstat_rwlock. XNU acquires these locks, not application code. configd (downstream victim): socket_common -> socreate_internal -> udp_attach -> nstat_udp_new_pcb -> lck_rw_lock_exclusivePossible enabling condition Matched 27 KDK disassembly shows a provider CLOSE path through cfil_sock_is_dead that directly marks the PCB DEAD and schedules GC without nstat cleanup. Public xnu-12377.121.6 (26.5) uses ordinary detach first. Provider CLOSE and final fd close are distinct. A DEAD PCB can retain an nstat locus/reference. Under _nstat_rwlock, nstat_idle_check removes the locus and releases the reference via in_pcb_checkstate(..., WNT_RELEASE, FALSE), which can take the socket mutex. Final close takes the reverse order; nstat_pcb_detach requests the nstat lock even when the locus is null. This is a possible enabling path from disassembly/source analysis, not proof of the preceding provider CLOSE in either incident. The older source already has opposite static lock orders; 26.5 is not proven immune. Questions for Apple Our provider calls closeReadWithError:nil and closeWriteWithError:nil when finished, as documented for NEAppProxyFlow. This does not validate the entire asynchronous lifecycle. Is this cycle known, and is there a fixed build? How is reverse locking avoided after the direct DEAD transition? Is there a supported workaround preserving flow cleanup? Close rate-limiting is not proven to eliminate the issue. Which diagnostics would help correlate provider CLOSE, DEAD, and final fd close? Attachments and timestamps Four EXCERPT attachments contain watchdog messages, raw frames, KDK annotations, and wait-owner records. Full reports are retained for private Feedback Assistant submission. 26A428_panic_excerpt.txt 26A428_configd_excerpt.txt 26A434_panic_excerpt.txt 26A434_configd_excerpt.txt 2026-10-15, +0800: configd capture -> embedded panic event: A: 04:24:43.1258 -> 04:26:44 B: 08:49:29.4349 -> 08:51:32 Use embedded panic times, not file/header timestamps. Original pairs match on boot identity; identifiers are omitted from excerpts. Sources: NEAppProxyFlow, XNU 26.5 comparison.

The environment: macOS 12.0 ~ 15.6 A NetworkExtension NEFilterDataProvider configured with filterSockets = YES, filterPackets = NO, and it doesn't actually block any network connection. QQMusic (download: https://y.qq.com/n/ryqq/download_detail/mac?ADTAG=YQQ) is constantly playing. Any of the following operations can reproduce the issue: Kill the NetworkExtension process and then restarted by the system. Disable the NEFilterDataProvider, and then enable it. When this problem occurs, there are two different phenomena on the NetworkExtension process: It is zombie, or is in high CPU state (100%). When the NetworkExtension process is zombie, obviously, the new network connections will enter it, and they can't be disposed by the old zombie process, so the network is disconnected. Spindump-qqmusic-ne-zombie When the NetworkExtension process is in high CPU state, its thread DispatchQueue "NEFilterExtensionProviderContext queue" is blocked in the kernel when calling close. Spindump-qqmusic-ne-cpuhigh In most cases, the network will recover after stopping QQ Music, that is the suspended zombie NetworkExtension process will exist or the cpu of it return to normal. To reproduce the issue in a simple environment, I have tried many ways to simulate the network behavior of QQMusic, but all failed. It seems that this issue is caused by UDP traffic of QQMusic, because everything is ok after blocking the UDP connections of QQMusic (the music is still playing at this time) in the NEFilterDataProvider.

When handleNewUDPFlow in NETransparentProxyProvider is used to handle UDP data from port 53, at the same time, run the script continuously to execute nslookup or dig, about tens of thousands of times later, the nslookup shows the error "isc_socket_bind: address not available". So I check the system port status, and find all of the ports from 49152 to 65535 are occupied. The number of net.inet.udp.pcbcount is also very high. net.inet.udp.pcbcount: 91433 Then I made the following attempts: handleNewUDPFlow function return false directly, the nslookup script runs with no problems. I write a simple network extension that use handleNewUDPFlow to reply the mock data directly, and only hijack the UDP data from my test program (HelloWorld-5555). My network exntension code: override func handleNewUDPFlow(_ flow: NEAppProxyUDPFlow, initialRemoteEndpoint remoteEndpoint: NWEndpoint) -> Bool { guard let tokenData = flow.metaData.sourceAppAuditToken, tokenData.count == MemoryLayout<audit_token_t>.size else { return false } let audit_token = tokenData.withUnsafeBytes { buf in buf.baseAddress?.assumingMemoryBound(to: audit_token_t.self).pointee } let pid = audit_token_to_pid(audit_token ?? audit_token_t()) if (!flow.metaData.sourceAppSigningIdentifier.starts(with: "HelloWorld-5555")) { return false } Logger.statistics.log("handleNewUDPFlow \(remoteEndpoint.debugDescription, privacy: .public) \(flow.hash), pid:\(pid), \(flow.metaData.sourceAppSigningIdentifier, privacy: .public)") flow.open(withLocalEndpoint: nil) { error in if let error { os_log("flow open error: %@", error.localizedDescription) return } flow.readDatagrams { data_grams, remote_endpoints, read_err in guard let read_data_grams = data_grams, let read_endpoints = remote_endpoints, read_err == nil else { os_log("readDatagrams failed") flow.closeReadWithError(nil) flow.closeWriteWithError(nil) return } let mockData = Data([0x01,0x02,0x03]) let datagrams = [ mockData ] guard let remoteEnd = remoteEndpoint as? NWHostEndpoint else { os_log("Not the NWHostENdpoint") flow.closeReadWithError(nil) flow.closeWriteWithError(nil) return } let endpoints = [ NWHostEndpoint(hostname: remoteEnd.hostname, port: remoteEnd.port) ] flow.writeDatagrams(datagrams, sentBy: endpoints) { error in if let error { os_log("writeDatagrams error: %@", error.localizedDescription) } os_log("writeDatagrams close") flow.closeReadWithError(nil) flow.closeWriteWithError(nil) } } } return true } My test program code: void send_udp() { int sockfd; struct sockaddr_in server_addr; char buffer[BUFFER_SIZE]; int bytes_sent; // create socket if ((sockfd = socket(AF_INET, SOCK_DGRAM, 0)) < 0) { perror("socket create failed"); exit(EXIT_FAILURE); } struct sockaddr_in local_addr; memset(&local_addr, 0, sizeof(local_addr)); local_addr.sin_family = AF_INET; local_addr.sin_addr.s_addr = htonl(INADDR_ANY); local_addr.sin_port = htonl(0); // bind if (bind(sockfd, (struct sockaddr*)&local_addr, sizeof(local_addr)) < 0) { printf("IPV4 bind errno:%d\n", errno); close(sockfd); return; } // server addr memset(&server_addr, 0, sizeof(server_addr)); server_addr.sin_family = AF_INET; server_addr.sin_port = htons(SERVER_PORT); server_addr.sin_addr.s_addr = inet_addr(SERVER_IP); // send & recv strcpy(buffer, "Hello, UDP server!"); bytes_sent = sendto(sockfd, buffer, strlen(buffer), 0, (struct sockaddr *)&server_addr, sizeof(server_addr)); if (bytes_sent < 0) { perror("sendto failed"); close(sockfd); exit(EXIT_FAILURE); } printf("sendto ok\n"); char recvbuf[128] = {0}; socklen_t len = sizeof(server_addr); int sz = recvfrom(sockfd, recvbuf, sizeof(recvbuf), MSG_WAITALL, (struct sockaddr *) &server_addr, &len); printf("recv sz: %d\n", sz); close(sockfd); return; } int main() { send_udp(); return 0; } 2.1 When I use bind in my program, after the program running tens of thousands of times, the ports are exhausted, and nslookup return the error "isc_socket_bind: address not available". The case looks like running the nslookup script, because the nslookup will call the bind. 2.2 When I remove the bind from my program, all the tests are go. I have made the above experiments on different systems: 13.x, 14.x, 15.x, and read the kernel source code about bind and port assignment, bsd/netinet/in_pcb.c bsd/netinet/udp_usrreq.c and find kernel will do different action for network extension by call necp_socket_should_use_flow_divert I have checked my network extension process by lsof and netstat, its sockets or flows are all closed properly. I don't know how I can avoid this problem to ensure my network extension to work long time properly. Apparently, the port exhaustion is related to the use of bind function and network extension. I doubt there is a port leak problem in system when use network extension. Hope for your help.