Thanks @eskimo for your insights, as always.
We use http://enet.bespin.org for reliable peer to peer communication, it's very old and very reliable UDP library, not dissimilar to QUIC - except that it existed way before QUIC. It provides multiple independent streams of data (reliable and unreliable) - on top of one UDP flow. It was created for multiplayer games and is super easy to work with.
I have a nice and small Swift based API written for that, in production on Linux and iOS/macOS since 2014 - that I plan to open source at some point.
I want to move it over to Network.framework - to make it work on the Apple Watch where BSD sockets do not work, and to gain performance by moving away from BSD sockets. Networking guys (from Apple) I had a chance to talk to claimed that moving to Network.framework will make everything faster...
I will try to explain as easy as possible what the problem is. Please bear with me :)
On each device you bind one UDP socket to a random port. And that socket/port is used for all outgoing and incoming communication with the other peers. The UDP datagrams contain a simple protocol that handles streams, retransmits, datagram re-ordering, etc.
So imagine App1 opens UDP Socket1, and binds it to ::Port1.
App2 opens Socket2, and binds it to ::Port2.
App3 open Socket3, and binds ::Port3.
Communication from App1 to App2 goes via local Socket1 leaving the machine on Port1 and going to App2 via Socket2 and remote Port2.
Communication from App1 to App3 goes also via local Socket1 leaving the machine on Port1 and going to App3 via Socket3 and remote Port3.
Communication from App2 to App3 goes also via local Socket2 leaving the machine on Port2 and going to App3 via Socket3 and remote Port3.
Replies the other way around.
In the BSD sockets world this is possible even on the same networking interface, and in the same UNIX process, because Socket1, Socket2, and Socket3 are independent file descriptors in the kernel and sendmsg and recvmsg can use these sockets to send to and receive from any UDP address. Via IIRC something that is called "unconnected UDP socket" mechanism.
And you essentially use the same fd for both sendmsg and recvmsg.
In the Network.framework world, Listener1 will allow me to receive UDP datagrams on Port1. Listener2 on Port2, and Listener3 on Port3.
Connecting from App1 (Port1) to App2 (Port2) is only possible by creating new NWConnection with requiredLocalEndpoint set to Listener1.requiredLocalEndpoint.
This works fine, as long as Port2 is on another networking interface or another machine.
But it breaks when Port1, and Port2 are used via NWListener or NWConnection in the same (UNIX) process and on the same networking interface. Say for example in one XCTestCase.
So if I want to create a peer1 (NWListener1) and peer2 (NWListener2) both bound to localhost, and then open a NWConnection1to2 it will break.
If I bind peer1 to en0 and the peer2 to lo0 it will work.
I have no insight into how is the Network.framework implemented, and also this only affects NWListeners and NWConnections using UDP, in the same process, and on the same interface. So not a real world use case, but only a testing use case.
Sorry for the long and chaotic description. I have a simple swift package demonstrating the code ready, and if you want I can open a DTS request to investigate this further.
At the moment I am proceeding in a way that for XCTestCase I create one peer using Network.framework and the other peer using BSD sockets and shuffle data between them.
But I would love to verify all the functionality when both peers use Network.framework. Which is easier over lo0.
I am thinking along the lines of enumerating all interfaces on the machine and binding peer1 to one interface (say en0) and the other to another interface (say en1) so that they can communicate together successfully.