Sending out to actual dest after the Packet intercepted by NEPacketTunnelProvider

As per : TN3120: Expected use cases for Network Extension packet tunnel providers | Apple Developer Documentation

It is clear that Packets that are read from NEPacketTunnelFlow are meant to be sent over a tunnel connection to a remote server for injection into a remote network. They are not meant to be dropped or re-injected back into the system.

In my usecase:

  • NEPacketTunnelProvider is separate process. which reads the packet using packetFlow.readPacketObjects

  • Send it over to other process i.e privileged helper(Non-bundle/command line tool/non sandboxed) via UDS IPC.

  • Helpers send to to remote tunnel and return back the packet to NEPacketTunnelFlow via same IPC.

  • NEPacketTunnelProvider uses packetFlow.writePacketObjects to inject packets.

Things works fine. We don't distribute it via Appstore.

We are now attempting to implement a on device bypass mechanism from helper tool side. Could you please suggest if there is any approach I could try, even if it involves proceeding at my own risk?

Answered by DTS Engineer in 856838022
instead of tunneling certain webstites to a remote VPN server, I want to allow those traffics to go directly

Yeah, we don’t support that. And by “we” I mean “DTS”, specifically. One of my erstwhile colleagues wrote TN3120 that talks about this in general terms, but I can be more specific here.

IMPORTANT The ‘tunnel what you claim’ issue described below is only one example of the problems you might encounter when you use a packet tunnel provider for unsupported things. There are others [1].

In general, Network Extension VPN providers (packet tunnel and app proxy) are expected to tunnel all traffic that they claim. So, in the case of a packet tunnel in destination IP mode [2], if your tunnel claims a destination network then it’s expected to forward traffic for that network through the tunnel.

DTS has, over the years, received a lot of cases like this, where folks claim a wide range of destination networks and then try to send some traffic through their tunnel and some traffic directly. This rarely ends well, and it’s not something we support.

This is mostly a problem on iOS, where folks really want to use a packet tunnel provider to bypass the privacy limitations imposed by other provider types. macOS is more forgiving on that front. In most cases you can get around this problem on macOS by moving up a layer to implement a transparent proxy.


In terms of unsupported techniques, I strongly recommend that you avoid trying to re-inject packets. I’ve never found a way to do that reliably.

One technique that typically works is to run your own TCP/IP stack and use that to turn the packets back into flows. You can then use standard networking APIs to proxy those flows. However, this approach has its own problems, which is why DTS doesn’t support it.

The ‘tunnel what you claim’ explanation above was largely copy’n’pasted from a response I sent on a recent DTS case, where the developer is doing just that and now they’ve stumbled into a horrible pitfall that causes things to fail badly in some very specific network environments.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] The other commons issue I see is folks running into problems trying to use VPN On Demand to implement Always-on VPN. That’s another thing that rarely ends well )-:

[2] As opposed to source application mode, which is one form of per-app VPN.

We are now attempting to implement a on device bypass mechanism from helper tool side.

What do you mean by “device bypass”?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

What do you mean by “device bypass”?

By on-device bypass I mean: when my NEPacketTunnelProvider intercepts packets, instead of tunneling certain webstites to a remote VPN server, I want to allow those traffics to go directly out through the device’s native network interface without going through the tunnel.

instead of tunneling certain webstites to a remote VPN server, I want to allow those traffics to go directly

Yeah, we don’t support that. And by “we” I mean “DTS”, specifically. One of my erstwhile colleagues wrote TN3120 that talks about this in general terms, but I can be more specific here.

IMPORTANT The ‘tunnel what you claim’ issue described below is only one example of the problems you might encounter when you use a packet tunnel provider for unsupported things. There are others [1].

In general, Network Extension VPN providers (packet tunnel and app proxy) are expected to tunnel all traffic that they claim. So, in the case of a packet tunnel in destination IP mode [2], if your tunnel claims a destination network then it’s expected to forward traffic for that network through the tunnel.

DTS has, over the years, received a lot of cases like this, where folks claim a wide range of destination networks and then try to send some traffic through their tunnel and some traffic directly. This rarely ends well, and it’s not something we support.

This is mostly a problem on iOS, where folks really want to use a packet tunnel provider to bypass the privacy limitations imposed by other provider types. macOS is more forgiving on that front. In most cases you can get around this problem on macOS by moving up a layer to implement a transparent proxy.


In terms of unsupported techniques, I strongly recommend that you avoid trying to re-inject packets. I’ve never found a way to do that reliably.

One technique that typically works is to run your own TCP/IP stack and use that to turn the packets back into flows. You can then use standard networking APIs to proxy those flows. However, this approach has its own problems, which is why DTS doesn’t support it.

The ‘tunnel what you claim’ explanation above was largely copy’n’pasted from a response I sent on a recent DTS case, where the developer is doing just that and now they’ve stumbled into a horrible pitfall that causes things to fail badly in some very specific network environments.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] The other commons issue I see is folks running into problems trying to use VPN On Demand to implement Always-on VPN. That’s another thing that rarely ends well )-:

[2] As opposed to source application mode, which is one form of per-app VPN.

@DTS Engineer Thanks for the detailed responses.

One technique that typically works is to run your own TCP/IP stack and use that to turn the packets back into flows. You can then use standard networking APIs to proxy those flows. However, this approach has its own problems, which is why DTS doesn’t support it.

Following questions around this:

  1. If I run my own TCP/IP stack in helper process and use it to send traffic directly to the outside network, will those packets still get intercepted by NEPacketTunnelProvider, given that the provider is currently configured to intercept everything? Running it inside extension process will work due to NECP policy considering Strict sandboxing of extension?
  2. Since you mentioned that this approach comes with its own problems, could you please outline some of them?
If I run my own TCP/IP stack in helper process … will those packets still get intercepted by NEPacketTunnelProvider …?

Yes. NECP acts to protect you from VPN loops but that only applies if you do the work in the process hosting your NE provider.

Since you mentioned that this approach comes with its own problems, could you please outline some of them?

I don’t have a comprehensive list. The nature of unsupported things is that they are… well… unsupported, so there no need for me track the problems in detail.

However, I can share a recent example. This is on iOS, and thus the helper process approach you’re using isn’t applicable. For reasons that are too complex to go into here, the packet tunnel provider is trying to force a specific connection to run over WWAN, and that’s being blocked by NECP. I’m not 100% sure if there’s a viable path forward for those folks. I’m beginning to suspect that they might need to rethink their overall design )-:

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@DTS Engineer I took following approaches to handle ICMP and UDP bypass:

  1. Create a raw socket at helper process side as this is not allowed from extension sandbox environment

  2. Upon receiving the pkt from extension over UDS at helper side, change source address of packets to en0 address and recommpute checksum.(source address will have utun address of Pkt Tunnel extn)

  3. Send packet on raw socket

    • it will be send to destinations.

    • Inbound traffics comes to kernel

    • ICMP gets deliver to applications. ICMP don’t require explicit binding by the app (system handles it), so raw injection + kernel delivery works naturally

    • Inbound UDP and TCP will not get deliver because kernel doesn’t know which socket to deliver it to

    • recvfrom for rawsocket doesn't works

  4. Handling Inbound UDP and TCP

    • sniff en0 using libpcap
    • Maintain some map / state table to pick only intrested packets
    • change the destination back to packet tunnel utun interface address and re compute checksum
    • tell packet tunnel to write it back

The final step in our inbound packet processing is failing intermittently. The kernel is silently dropping the packet sometime during the packetFlow.writePacketObjects call, and it never reaches the application. How can we debug why the kernel is dropping it?

My experience is that, when folks ignore the advice in TN3120, they run into a wide range of problems. Moreover, the exact set of problems can change between OS releases. Such is the nature of unsupported thing. Given that, I can’t help you go further down this unsupported path.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Sending out to actual dest after the Packet intercepted by NEPacketTunnelProvider
 
 
Q