Ok, I am going to say that is horribly unclear then -- I read it completely differently, and "the file descriptor should not be closed by the application until the cleanup handler is called" would have made all the difference... 😄 And now I can't get it to call the cleanup handler at all, sigh.

The basic flow of the code is: Create a socketpair Create a DispatchIO using socket[0] Create a FileHandle using socket[1] Do work When done, DispatchIO.close() and FileHandle.close() Also close the sockets using Darwin.close(). CRASH! The problematical one seems to be the file descriptor for the DispatchIO.

So, yay, I can reproduce it in my test code. And I used fdguard (with CLOSE not DUP), and never unguard the file descriptors, and it still gets the EV_VANISH crash.

The crashed thread is Thread 5 Crashed: 0 libdispatch.dylib 0x7ff80af41e5f _dispatch_source_merge_evt.cold.1 + 24 1 libdispatch.dylib 0x7ff80af25bb1 _dispatch_source_merge_evt + 149 2 libdispatch.dylib 0x7ff80af2f9b6 _dispatch_event_loop_merge + 112 3 libdispatch.dylib 0x7ff80af21db3 _dispatch_workloop_worker_thread + 438 4 libsystem_pthread.dylib 0x7ff80b0c5fd0 _pthread_wqthread + 326 5 libsystem_pthread.dylib 0x7ff80b0c4f57 start_wqthread + 15 I've attached a trimmed & edited copy of the full crash log (but not the entire .ips file). crash.txt

FB14453214, alas. I'm going to try my minimal TPP as well.

Here is the (somewhat edited) sequence of system logs for a flow. I pared down the excludedRules a bit, and added: let appleRule = NENetworkRule(destinationNetwork: NWHostEndpoint(hostname: "17.0.0.0", port: "0"), prefix: 8, protocol: .any) so nothing in 17/8 should have come to the provider, right? And yet it did. I must be doing something very very wrong, but I can't figure it out yet. com.kithrup.SimpleTPP.Provider: (libnetworkextension.dylib) [com.apple.networkextension:] (0): Flow 1864790775 is connecting com.kithrup.SimpleTPP.Provider: (libnetworkextension.dylib) [com.apple.networkextension:] (1864790775): New flow: NEFlow type = datagram, app = com.apple.identityservicesd, name = , 192.168.43.105:16393 <-> 17.252.14.81:3478, filter_id = , interface = en0(bound) com.kithrup.SimpleTPP.Provider: (NetworkExtension) [com.apple.networkextension:] [Extension com.kithrup.SimpleTPP.Provider]: Calling handleNewUDPFlow with UDP com.apple.identityservicesd[{length = 20, bytes = 0x9369f2790daea880d6d0dca66519aa9ae04f7bcc}] local port 16393 interface en0(bound), remoteEndpoint = 17.252.14.81:3478 com.kithrup.SimpleTPP.Provider: [com.kithrup.SimpleTPP.Provider:redirector] handleNewUDPFlow(_:initialRemoteEndpoint:): new UDP flow for host 17.252.14.81:3478 app com.apple.identityservicesd com.kithrup.SimpleTPP.Provider: [com.kithrup.SimpleTPP.Provider:redirector] Got flow for com.apple.identityservicesd identityservicesd: (libquic.dylib) [com.apple.network:quic] quic_crypto_new_flow [C382:2] [b3582e00-b4c3556dcba3be09] TLS stream is: [C383] identityservicesd: (Network) [com.apple.network:connection] [C383 C99F226F-F5A6-49B2-9AE1-ACE580B7619A IPv4#c62ff9e2:3478 quic, tls, definite, attribution: developer, reuse local address, context: IDSRealTime (private), proc: 24A8D0C7-FB75-37DE-8065-5EB68A7790DE, effective proc: 7FD7A321-FDAF-3CF4-926E-BF555C540CE0, local address: 192.168.43.105:16393, required interface: en0(13), has demux] start identityservicesd: (Network) [com.apple.network:connection] [C383 IPv4#c62ff9e2:3478 initial socket-flow (satisfied (Path is satisfied), interface: en0[802.11], scoped, ipv4, ipv6, dns, uses wifi)] event: path:start @0.000s identityservicesd: (Network) [com.apple.network:connection] [C383 IPv4#c62ff9e2:3478 waiting socket-flow (satisfied (Path is satisfied), interface: en0[802.11], scoped, ipv4, ipv6, dns, uses wifi)] event: path:satisfied @0.000s, uuid: A457D7C0-8173-4B04-B722-747C0287E464 identityservicesd: (Network) [com.apple.network:connection] [C383 IPv4#c62ff9e2:3478 in_progress socket-flow (satisfied (Path is satisfied), interface: en0[802.11], scoped, ipv4, ipv6, dns, uses wifi)] event: flow:start_connect @0.000s identityservicesd: (Network) [com.apple.network:connection] nw_connection_report_state_with_handler_on_nw_queue [C383] reporting state preparing identityservicesd: (Network) [com.apple.network:connection] nw_flow_connected [C383 IPv4#c62ff9e2:3478 in_progress socket-flow (satisfied (Path is satisfied), interface: en0[802.11], scoped, ipv4, ipv6, dns, uses wifi)] Joined protocol connected (quic) identityservicesd: (Network) [com.apple.network:connection] [C383 IPv4#c62ff9e2:3478 in_progress socket-flow (satisfied (Path is satisfied), interface: en0[802.11], scoped, ipv4, ipv6, dns, uses wifi)] event: flow:finish_transport @0.000s com.kithrup.SimpleTPP.Provider: [com.kithrup.SimpleTPP.Provider:redirector] We are bypassing the app com.apple.identityservicesd (path /system/library/privateframeworks/ids.framework/identityservicesd.app/contents/macos/identityservicesd)! com.kithrup.SimpleTPP.Provider: (NetworkExtension) [com.apple.networkextension:] [Extension com.kithrup.SimpleTPP.Provider]: provider rejected new flow UDP com.apple.identityservicesd[{length = 20, bytes = 0x9369f2790daea880d6d0dca66519aa9ae04f7bcc}] local port 16393 interface en0(bound) identityservicesd: (libboringssl.dylib) [com.apple.network:boringssl] boringssl_session_apply_protocol_options_for_transport_block_invoke(2007) [C383:1][0x125f17eb0] TLS configured [min_version(0x0304) max_version(0x0304) name(redacted) tickets(true) false_start(true) enforce_ev(false) enforce_ats(false) ech(false)] containermanagerd: (ContainerManagerCommon) [com.apple.containermanager:fs] stat [<private>]: exists: 1, isDirectory: 0, fsNode: <~~~> kernel: (1864790775): No more valid control units, disabling flow divert kernel: (1864790775): Skipped all flow divert services, disabling flow divert identityservicesd: (libboringssl.dylib) [com.apple.network:boringssl] boringssl_context_info_handler(2133) [C383:1][0x125f17eb0] Client handshake started com.kithrup.SimpleTPP.Provider: (libnetworkextension.dylib) [com.apple.networkextension:] (1864790775): Closing reads (sending SHUT_WR), closed by plugin (flow error: 0) com.kithrup.SimpleTPP.Provider: (libnetworkextension.dylib) [com.apple.networkextension:] (1864790775): Closing writes, sending SHUT_RD com.kithrup.SimpleTPP.Provider: (libnetworkextension.dylib) [com.apple.networkextension:] (1864790775): Destroying, client tx 0, client rx 0, kernel rx 0, kernel tx 0 identityservicesd: (libboringssl.dylib) [com.apple.network:boringssl] boringssl_context_info_handler(2150) [C383:1][0x125f17eb0] Client handshake state: TLS client enter_early_data

Do you mean that when you have the Proxy Provider running that FaceTime is not working or FaceTime does not work at all? When it is running, neither FaceTime video nor audio work. I can get it to connect outbound (haven't tried inbound recently, so I forget), but there's just no audio or video. Let me get all of the log messages for a small time frame.

Well, I can do the same thing in Swift, it just wasn't as obvious, by using #if and typealias appropriately. It's not much uglier than in C, so that's fine. The goal is to test things like error states, open/close states, and various callbacks. So I mocked up my own NWHostEndpoint (aka TestNWHostEndpoint), NEAppProxyFlow, etc. They're immensely simplistic, and won't test as well as the real thing, but I should be able to test my API fairly well. (This was all motivated by the fact that my attempt to use DispatchIO crashes, every time, with enough load, and I can't figure out why. So maybe if I mock up a couple hours' worth of 100+ flows/sec, I might be able to track it down.)

Matt Easton over at https://forums.developer.apple.com/forums/thread/756508 said otherwise -- specifically, For example, your flow copying logic relies on NEAppProxyFlow objects, but you could mock up a test object that represents the infrastructure in this class to just pipe network traffic into it to exercise your flow copying logic. If it were in ObjC, I could do #if and/or #define tricks to include different headers, but I don't know how I'd do that with Swift. (And I can never remember how to define something in build settings or the command line and have Swift able to use #if for it, so I always have to go searching for it again.)

Ok, that is a dispatch thread limit; that's different from OS threads. That makes sense.

Alas. I did figure out my design problem, which was creating all of the threads, however, the user-space proxy has a separate thread per network connection.

Fortunately that case didn't come into play -- I only .activate() and .cancel() it. The documentation wasn't really clear, but I assume a cancelled source can be released. Also, as I said, that didn't appear to be my real problem, so I'm not sure what all was going on. I've rewritten the code that uses kevent into ObjC, to see if that helps (ie., I got very confused about conversions again).

Oh, hm. Ok, that's not it. It's when my object is deinit'ted at all. That seems very weird.

BTW: the forum software things that the abort cause number is "sensitive language," and it's taken me about 20 minutes to figure tha tout.

What changes were made to NWEndpoint? I filed FB14076199