Transparent Proxy Providers and networking

I think I am coming in with some missing context, but why is it needed to proxy traffic locally over a socket? Are you able to proxy traffic from the actual TPP if you needed to using a modern API such as NERelay?

Matt Eaton - Networking

I'm not sure how NSRelay works. (Oh! It's the same thing our user-land proxy does, I think. I'm talking about the problems from the TPP.)

Our TPP talks to a user-land proxy that does a bunch of stuff. It's cross-platform; for Windows, it is a much simpler design. The XPC data interchange (flow.read() -> XPC call -> user-land agent -> process, get response -> XPC call -> flow.write()) has been fraught. I am trying to see if instead of XPC, I can connect to the user-land proxy's proxy port.

And, well, I can -- but after the first exchange of data (TPP: socket -> connect -> write -> read -> write; agent socket -> bind -> listen -> accept -> read -> write -> read), the data that the TPP writes ... doesn't show up on the interface, and isn't received by the user-land agent.

(We're not doing just port 80 and 443.)

Okay, thank you for that background. Again, probably coming in late to the conversation here, but what were the issues that you had with XPC?
Also, is there a reason you cannot build the cross platform components to your library to work directly with your TPP provider also? For example, build a static library that links with your provider and builds cross platform? That way you would not need to call out to any other process?

Matt Eaton - Networking

A bunch of issues, but that's not the problem I'm trying to solve right now -- right now, I'm trying to understand why the 2nd write() doesn't show up on the interface (according to tcpdump, and according to the socket's other side's read()).

The XPC issues have mostly to do with the fact that macOS is, as is sadly usual, an also-have, and the bulk of the code is built around 3rd party libraries that use select/kevent, and hooking them up to the XPC model is, as I said, fraught. So I am trying some other approaches.

right now, I'm trying to understand why the 2nd write() doesn't show up on the interface

and hooking them up to the XPC model is, as I said, fraught. So I am trying some other approaches.

In general, your socket issues sound like a routing table issue but it's hard to clearly tell what's going on there. To debug this further on your end I would look a what is happening to your socket with an application like nettop to see if its changing states during the connection / request action. Also, try checking out the routing table to see if that gives you any more info about where your socket is being routed. netstat -rn.

Having said all of that above, I will leave you to debug what is happening with the socket / proxy case as I do not recommend this approach. If you need additional help here I would be glad to continue the conversation regard XPC or linking your existing library with the TPP.

Matt Eaton - Networking

The problem with saying it's the routing table is that the first cycle -- writing to the socket from the TPP, reading from the socket in the user-space proxy, then reading that from the socket in the TPP -- works; it's the second write from the TPP that ... vanishes. The connection is localhost:12345, and running tcpdump -i lo0 -s 0 -vvvvvvv -A port 12345 shows the first two, but the last data ... isn't there.

The write() succeeds -- it returns the number of bytes, and the addresses on both ends of the socket are correct.

AH HA!

Honestly, I have no idea if anyone cares about this, but I'm going to document it anyway.

First, I verified that the problem was with my code by writing a very simple proxy. (I still can't see how one uses NERelay!) After all weekend working on it, I got it all working. I cheated and used ObjC for the proxy code, because doing things with sockets was easier.

Second: having my simple TPP app successfully using a proxy, I took the changes I'd made to my code for the provider over to our real app, and it immediately continued to not work.

After days of working on that, the bulk of my problems were due to my Swift clumsiness.

In one case:

let nwritten = data.withUnsafeBytes { (ptr: UnsafeRawBufferPointer) in
    var t = ptr
    let count = data.count
    let kr = Darwin.write(self.socket, &t, count)
    return kr
}

That did a great job of writing the UnsafeRawBufferPointer to the socket, when what I wanted was, obviously, the data that it pointed to.

Similarly, in another place, I had

var buffer = message.data(using: .ascii)!
let nwritten = Darwin.write(self.socket, &buffer, buffer.count)

and yeah that wasn't right. I changed them both to be variants of

let nwritten = buffer.withUnsafeBytes { ptr in
    return Darwin.write(self.socket, ptr, buffer.count)
}

and then AMAZINGLY it all started working!

Glad you found a path forward here and thank you for adding your follow-up solution. As a final note, you may not need need to explicitly call Darwin.write if that module is imported.

Matt Eaton - Networking

At one point, I had an instance variable named socket, and a function called write(), so I got into the habit, for that file, of using Darwin. for everything. 😄