QNE2TransparentProxyMac sample code

QNE2TransparentProxyMac isn’t official sample code. Rather, it’s a test project that I created that I’ve given out to a few developers on a one-to-one basis.

Most of my QNE test projects don’t contain any real code. Rather, they’re primarily focused on packaging issues. That’s very much the case for QNE2TransparentProxyMac. It doesn’t actually do any networking, it just shows how to get a transparent proxy to the point where the various handle-new-flow methods get called.

And these days that’s not so valuable because Xcode has move forward to the point where the NE packaging issues that caused a bunch of grief during the early days of NE have all been resolved.

If you have specific questions about transparent proxies, I’m happy to answer them here.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thanks for getting back to me. I'm afraid my poor editing above confused the issue. The project claims it is based on the QNE2TransparentProxyMac and I've been working on it for a while to finish TLS inspection and filtering. The previous developer had moved on before I joined the project so I don't have whatever code he might have received.

I want to add support for UDP and noticed it's not enabled in the Network Extension and there's no UDP flow copier. Is there some transparent proxy example that includes support for UDP?

I'm following the TCP example to implement UDP support but would prefer not to reinvent it if it already exists. Thanks.

• Peter

In func handleNewTCPFlow(_ flow: NEAppProxyTCPFlow) we create a connection as:

let connection = NWConnection(to: flow.remoteEndpoint.nwEndpoint, using: .tcp)

When converting this to UDP, the NEAppProxyUDPFlow doesn't have a flow.remoteEndpoint so it's unclear to me where I should get this from. Presumably each datagram has a destination which defines which flow it belongs to.

Is there some transparent proxy example that includes support for UDP?

No.

Notably, QNE2TransparentProxyMac doesn’t include a TCP flow copier. Its handle-new-flows methods all return false. I suspect your erstwhile colleague got that code from elsewhere. It might’ve been from Handling Flow Copying.

Presumably each datagram has a destination which defines which flow it belongs to.

Right.

This is confusing because we’re a bit inconsistent about the meaning of the word flow:

• Normally I use the term UDP flow to represent a sequence of datagrams with the some local IP / local port / remote IP / remote port tuple.

• However, that’s not what flow means in the case of NEAppProxyUDPFlow. Rather, it represents a UDP network ‘handle’ used by an app. For example, if the app is using BSD Sockets, the NEAppProxyUDPFlow is equivalent to the UDP socket file descriptor.

So, an NEAppProxyUDPFlow object constrains the local IP / local port part of the tuple but not the remote IP / remote port part. That can vary on a datagram-by-datagram basis. That’s why the various readDatagrams(…) methods return the outgoing datagrams and their associated endpoints.

Note that it is possible for the NEAppProxyUDPFlow object to constrain the remote IP / remote port part as well. That’s what you get when, for example, you use BSD Sockets to create a connected UDP socket. In that case the endpoint will come to you via the initialRemoteFlowEndpoint parameter of the -handleNewUDPFlow:… method.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I’m still debugging my UdpFlowCopier and could use some guidance as some aspects are confusing.

I need to copy datagrams between a provider flow object and an actual UDP socket or “connection”. Can you help me understand the difference between a NetworkExtension.NWEndpoint and Network.NWEndpoint and where I would need to use one versus the other?

Following the TCP example, the PassThroughProviderCore calls handleNewUDPFlow(_ flow: NEAppProxyUDPFlow). There’s no remote endpoint but I can retrieve the local endpoint as

guard let nwEndpoint = flow.localEndpoint?.nwEndpoint else { return }

I’m trying to open a UDP listener on this endpoint as follows but it mostly returns failed to create listener on port 0

private func handleStart() -> State {

    let port = portForEndpoint(self.localEndpoint) ?? 0
    logger.debug("UDPFlowCopier - handleStart copier \(self.osLogID) port \(port.rawValue, privacy: .public)")
    let params = NWParameters.udp
    params.allowFastOpen = true
    self.listener = try? NWListener(using: params, on: port)
    self.listener?.stateUpdateHandler = { update in
        switch update {
        case .ready:
            self.isReady = true
            //self.processEvent(.didOpenConnection)
        case .failed, .cancelled:
            // Announce we are no longer able to listen
            self.listening = false
            self.isReady = false
                logger.debug("UDPFlowCopier - copier \(self.osLogID) failed to create listener on port \(port.rawValue, privacy: .public)")
            self.stop()
        default:
            ()
        }
    }
    self.listener?.newConnectionHandler = { connection in
        self.createConnection(connection: connection)
    }
    self.listener?.start(queue: self.queue)

    return .openingConnection
}

Can you offer any insight? Thanks

Can you help me understand the difference between a NetworkExtension.NWEndpoint and Network.NWEndpoint and where I would need to use one versus the other?

Sure. And that is, indeed, a gnarly edge case.

NetworkExtension.NWEndpoint is an Objective-C class. It was introduced prior to the introduction of Network framework proper. Network framework is a Swift API [1] and, in Swift, it makes sense to represent endpoints as an enum, so Network.NWEndpoint is an enum.

Quoting TN3151 Choosing the right networking API:

Network Extension in-provider networking includes NWUDPSession. While there are some very limited circumstances where this is still useful, in most cases it’s better to use Network framework. For more details, see In-Provider Networking.

So, what are those circumstances? In short:

• If your product supports system prior to the introduction of Network framework, that is, macOS 10.14. That’d be pretty unusual these days.

• There are a few Network Extension APIs that work in terms of NetworkExtension.NWEndpoint. In macOS 15 we added parallel mechanisms to that support Network.NWEndpoint. If you support systems prior to that, you’ll need to either use the in-provider network APIs or implement some sort of shim to get things working on top of Network framework.

The additional of these parallel mechanisms was tricky, involving some exciting use of the .apinotes file [2].

I’m trying to open a UDP listener on this endpoint

Wha? I can’t see any circumstances where that’d be necessary.

I need to copy datagrams between a provider flow object and an actual UDP socket or “connection”.

Doing that with NWConnection is tricky due to a semantic disparity. There isn’t necessarily a one-to-one mapping between NEAppProxyUDPFlow and NWConnection because:

• NEAppProxyUDPFlow can represents a single local endpoint that sends to multiple remote endpoints.

• NWConnection represents a UDP flow, that is, a series of datagrams that all share the same local IP / local port / remote IP / remote port tuple

I see two paths forward:

• Use a different API. Notably, BSD Sockets supports the same model as NEAppProxyUDPFlow.

• Create a different NWConnection for each remote endpoint.

I’m not entirely sure whether the NWConnection option will work though, because I don’t see a good way to create multiple outgoing connections with the same local port [3]. I’m still digging into that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] It also has a low-level C API.

[2] Below is an explanation of this I wrote in another context. It talks about NWPath, not NWEndpoint, but the same process applies for both.

In Swift 6 mode, NetworkExtension.NWPath is published as NetworkExtension.__NWPath so that Network.NWPath can take priority. It does this by API notes. Specifically, NetworkExtension.framework/Versions/A/Headers/NetworkExtension.apinotes has this:

…
- Name: NWPath
  SwiftPrivate: true
…
SwiftVersions:
- Version: 5.0
  …
  - Name: NWPath
    SwiftPrivate: false
  …

In the Swift 6 language mode, SwiftPrivate is true and thus the type gets published as __NWPath. In Swift 5 language mode, SwiftPrivate is false and thus it continues to be published as NWPath.

[3] The obvious option, setting requiredLocalEndpoint, just generates an EADDRINUSE, at least in my case.

So I have developed a UDP flow copier using BSD Sockets (via GCDAsyncUDPSocket) which appears to be sending and receiving UDP datagrams but I am struggling to get DNS querries to resolve. I can see in Wireshark UDP DNS queries are not getting out to the wire.  Internal logging suggests packets written to a normal UDP socket are being intercepted by our transparent proxy (SysEx) and routed back to our app. Do I need to use an In-Provider API or respect some property to send packets to their actual destination? The TCP Flow Copier uses an NWConnection which doesn’t fit the UDP flow semantics.

One idea I tried exploring is to have my flow copier not handle DNS or broadcast traffic but I don’t see an easy way to do this.

There are two ways to control what flows you want to handle. One is the provider settings that let you specify an array of included and excluded NENetworkRules. The excluded networks take precedence so if you exclude a network it will bypass your proxy regardless of any included networks. When I try to exclude traffic for port 53 (DNS) I am confronted with the following error message:

startProxy - did not start, error: NETunnelProviderErrorDomain / 1 
53 cannot be specified as the port for transparent proxy network rules. 
Create a NENetworkRule object with a domain endpoint to divert DNS queries for that domain to the provider.

If the intent of not allowing port 53 is to avoid proxying all DNS traffic the restriction seems to be preventing the result it was intended to achieve. The other way to control what flows you want to handle is in the handleNewFlow method:

override func handleNewFlow(_ flow: NEAppProxyFlow) -> Bool

but this also fails to meet the need because FOR NEAppProxyUDPFlow there is no remote endpoint information available.

Still looking for a workable path or example.

I found a way to not proxy DNS using the recently deprecated: func handleNewUDPFlow(_ flow: NEAppProxyUDPFlow, initialRemoteEndpoint remoteEndpoint: NWEndpoint) -> Bool

which provides the initial remote endpoint. Investigating NWUDPSession for the UDP flow copier.

I’m glad to hear you’re making progress.

using the recently deprecated

Just to be clear, -handleNewUDPFlow:initialRemoteEndpoint: was deprecated in favour of -handleNewUDPFlow:initialRemoteFlowEndpoint:. The only difference is that the former uses the old in-provider networking NWEndpoint Objective-C type, whereas the latter uses Network framework’s nw_endpoint_t.

The Swift projection of nw_endpoint_t is NWEndpoint, which confuses things because it has the same name as the old Objective-C type. To opt in to the new method in Swift, adopt the NEAppProxyUDPFlowHandling protocol. For example:

class MyProvider: NETransparentProxyProvider, NEAppProxyUDPFlowHandling {

    func handleNewUDPFlow(_ flow: NEAppProxyUDPFlow, initialRemoteFlowEndpoint remoteEndpoint: NWEndpoint) -> Bool {
        return false
    }
}

Note that the NWEndpoint in this snippet is the Network framework Swift type. You can replace it with the qualified type, Network.NWEndpoint, and the code still compiles.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I still seem to be missing something. I updated my UDP flow copier to use an NWUDPSession based on the initial remote endpoint. When I try to send a datagram to the actual remote destination, it's somehow being intercepted by my proxy (as if originating from an app) instead of going out over the wire. The send is a simple call like this:

udpSession?.writeDatagram(messageData)

I'd welcome any ideas for what might be missing or what to try next. Thanks!

I've been assuming the network extension mechanism is aware of the network stack order so that when I send a UDP datagram from within my extension it enters the outbound network stack below my extension. Perhaps this is an unwarrented assumption dating back to the STREAMs network plugin architecture. Do I need to tag or otherwise identify my outgoing UDP datagram(s) so I can ignore them in a subsequent call to handleNewUDPFlow(_ flow:)?

I am looking for help debugging a UDP Flow Copier in the context of a TransparentProxyProvider Network System Extension. My SysEx has been handling TCP flows reliably for months but extending it to handle UDP flows including DNS has been problematic. A detailed problem description is attached.

I am prepared to open a DTS support incident (on request) but do not have a "focussed example project" since testing System Extensions is not easily focussed into a small example. The problem is consistently reproducible per the detailed logging provided.

Please see the attached problem description and diagram.

Thank you

When I try to send a datagram to the actual remote destination, it's somehow being intercepted by my proxy … instead of going out over the wire.

Hmmm, that’s weird. That sort of loopback is exactly what NECP should prevent. See A Peek Behind the NECP Curtain.

I’ve done this [1] with a DNS proxy and didn’t encounter this problem. However, DNS proxies and transparent proxies are only similar, they’re not exactly the same, so there’s no guarantee that my experience applies in this case.

I have one thing I’d like you to test here: Apply the flow’s metadata to your connection. You can only do this if you’re using NWConnection, so you’ll need to switch to that for the sake of this test.

IMPORTANT Don’t spend a lot of time on this. Rather, just hack something together for the benefit of this test. If it helps, we can then talk about how best to proceed.

So apply the metadata, do this:

let flow: NEAppProxyFlow = …
let parameters: NWParameters = …
if #available(macOS 15.0, *) {
    flow.setMetadata(on: parameters)
} else {
    fatalError()
}

then create your connection from the parameters.

Note setMetadata(on:) is only available on macOS 15, and I suspect your product needs to support earlier systems. Hence my comment about hacking something together just for this test.

Does that help with the loopback issue?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Albeit not exactly this. I was using NWConnection rather than NWUDPSession. I was able to do that because I was routing all traffic to a specific DNS server. And this is a few years ago, so something could’ve come unstuck in the interim.

Per the updated description the packets are getting out to the wire. The problem I'm still seeing is with writeDatagrams() back to the original flow that's disabling my TransparentProxyProvider. I've included a bit of detail including the line of code that's causing the failure.

Scouring the system log I found this: default 15:48:52.258383-0400 DTNetworkTracker Last disconnect error for DTNetworkTracker changed from "The VPN session failed because an internal error occurred." to "none".

My question now is what's causing this failure or how to write datagrams back to the original NEAppProxyUDPFlow flow safely?

Thanks!

Ah, sorry, I missed a key point of state in your previous response. Hiding all the important stuff in a text attachment is… well… innovative )-:

The VPN session failed because an internal error occurred.

Hmmm, that’s weird, and also not good.

Writing datagrams to the flow shouldn’t be hard. A message like this:

IPCConnection Starting XPC listener for mach service

suggests that something crashed and had to be restarted. Such a failure would also be a good candidate for the internal error quoted above.

Do you see any crash reports generated? Either for your sysex or for various NE helper processes?

If you see one, please post it here. See Posting a Crash Report for advice on how to do that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"