Hi,
Our project is a MacOS SwiftUI GUI application that bundles a (Sandboxed) System Network Extension, signed with a Developer ID certificate for distribution outside of the app store. The system network extension is used to write a packet tunnel provider (NEPacketTunnelProvider), as our project requires the creation of a TUN device.
In order for our System VPN to function, it must reach out to a (self-hosted) server (i.e. to discover a list of peers). Being self-hosted, this server is typically not accessible via the public web, and may only be accessible from within a VPN (such as those also implemented using NEPacketTunnelProvider, e.g. Tailscale, Cloudflare WARP).
What we've discovered is that the networking code of the System Network Extension process does not attempt to use the other VPN network interfaces (utunX) on the system. In practice, this means requests to IPs and hostnames that should be routed to those interfaces time out. Identical requests made outside of the Network System Extension process use those interfaces and succeed.
The simplest example is where we create a URLSession.downloadTask for a resource on the server. A more complicated example is where we execute a Go .dylib that continues to communicate with that server. Both types of requests time out.
Two noteworthy logs appear when packets fail to send, both from the kernel 'process':
cfil_hash_entry_log:6088 <CFIL: Error: sosend_reinject() failed>: [30685 com.coder.Coder-Desktop.VPN] <UDP(17) out so b795d11aca7c26bf 57728068503033955 57728068503033955 age 0> lport 3001 fport 3001 laddr 100.108.7.40 faddr 100.112.177.88 hash 58B15863
cfil_service_inject_queue:4472 CFIL: sosend() failed 49
I also wrote some test code that probes using a UDP NWConnection and NWPath availableInterfaces. When run from the GUI App, multiple interfaces are returned, including the one that routes the address, utun5. When ran from within the sysex, only en0 is returned.
I understand routing a VPN through another is unconventional, but we unfortunately do need this functionality one way or another. Is there any way to modify which interfaces are exposed to the sysex?
Additionally, are these limitations of networking within a Network System Extension documented anywhere? Do you have any ideas why this specific limitation might exist?
Apple systems have a mechanism called NECP that manages access to network interfaces. The VPN subsystem uses that to try to prevent VPN loops. I have some basic info about this A Peek Behind the NECP Curtain.
You might be able to get around this by binding your connections to the specific interface you want to run over. You can’t do that with URLSession, but I presume your Go code is using BSD Sockets under the covers and it’s definitely possible there. I talk about these ideas in the various posts hung off Extra-ordinary Networking.
Please try this out and let me know how you get along. Honestly, I’m not sure that this’ll work, but it’s worth exploring before you attempt anything more convoluted.
Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"