I'm reading through https://developer.apple.com/forums/thread/767467 now

I need to use Bonjour because a big selling point of the app im working on is that N devices will be able to discover and connect to each other while offline (ie no wifi network, ie out in the middle of no where). The devices only need to have their wifi toggle on.

Hello, I'm back revisiting this and have some findings to report. When I tried to create a group as seen below let descriptor = NWMultiplexGroup(to: endpoint) guard let identity = networkingService.importIdentityFromPKCS12() else { print("[browseResultsChangedHandler] could not import identity") continue } do { let parameters = try NWParameters(identity: identity) let group = NWConnectionGroup(with: descriptor, using: parameters) group.stateUpdateHandler = { _ in } group.newConnectionHandler = { _ in } group.start(queue: .global()) networkingService.addGroup(deviceID, group) } catch { } Note that endpoint comes from browser's browseResultsChangedHandler Identity comes from me importing a SecIdentity from the user's p12 Parameters has the identity in it so that QUIC works. My app crashes when it tries to create the group and I see the following error nw_group_descriptor_allows_endpoint Endpoint 1B1654DF-62B5-4B29-9495-C4965E8E99FF._captadoh._udp.local. is of invalid type for multiplex group -[NWConcrete_nw_group_descriptor initWithType:member:groupID:] Invalid endpoint type specified for group descriptor of type multiplex Does this mean that a peer to peer endpoint not work with the multiplex group?

Again, QUIC allows for a bunch of designs, and you have to choose one. Could you share the designs?

And just like how we determined for the non NWConnectionGroup setup.... We don't want Device A discovering and connecting to Device B and also Device B discovering and connecting to Device A. We want to have one side (via the comparison of device IDs) discover and connect to the other so we wont have multiple connection groups. I think for me this is the final detail that a tad fuzzy. Could you walk through an example of three devices, Device A, B and C? Sharing what you would save[1] on each device? [1] saving things like the connection group, each connection or endpoints?

Thank you for the reply. Now for my sanity check... Client -> NWBrowser Server -> NWListener Connection -> NWConnection In the NWBrowser I discover NWEndpoints With a discovered endpoint I can create a connection group (ie if I discovered 5 endpoints, excluding my own, I would then create 5 connection groups) From a single connection group I can create many connections to the endpoint that was used to create the connection group. When I create the connection group in the client I need to set newConnectionHandler to handle new incoming connections. In NWListener I listen for new connection groups. I do this by setting newConnectionGroupHandler. For each group on the server i setup newConnectionHandler to handle new connections. When everything is said and done and properly setup: a device will have a list of groups (a group for each device it has discovered) a device can have a list of connections that came from a group In my head I see: Client client discovers endpoints client creates connection groups client setup connection group to listen for new connections client can create new connections from a connection group client is saving the connection group and the many connections Server server listens for new connection groups server handles new incoming connection groups server setup connection group to listen for new connections service is saving the connection group and the many connections Questions: What am I gaining from using a connection group vs how I was doing things before with just NWConnections? Is the benefit of using a connection group is that both sides (the server and the client) have a connection group so either one could simply create a new connection(ie let connection = NWConnection(from: group)) from the group and send data? See below for design plan / question Design Use comparison of peer IDs to determine who initiates connection to who. So if this device's ID (this device will be an iPhone) is greater than the device of the discovered peer ID (the discovered device will be an iPad) then the iPhone will: create a connection group from iPad endpoint create a connection from the group to connect to the discovered peer iPad the discovered peer iPad will then receive a new group connection the discovered peer iPad will setup the group connection the discovered peer iPad will receive a new connection on the connection group In the end the iPhone will have a connection group and a single connection and the iPad will have a connection group and a single connection. Either side (iPhone or iPad) can easily create a new connection from their group connection. If they did, in our example, both sides would still have a single connection group but two connections now. Is my thinking correct?

Hello - I've been thinking... should I even be using QUIC? If I need to use UDP to live stream video then why not use TCP for sending simple messages (ie "stop video recording", "take photo", "send me your latest media") and downloading files from devices? If I went to TCP I would be able to use PSK and it would allow me to get rid of a whole piece of my backend (generating the identity). You've been on this journey with me for a while (my first post - https://developer.apple.com/forums/thread/768961). I do value your expertise. I remember you sharing QUIC would be the route to go but i think you shared that before you found the bug that restricted QUIC from being used in a best effort way. I do appreciate you taking the time to work through this with me.

Thank you! Would you be able to provide a simple example on how to use NWConnectionGroup with QUIC and peer to peer and star architecture? Im confused on what to put for the with param in the NWConnectionGroup initializer. https://developer.apple.com/documentation/network/nwconnectiongroup/init(with:using:) I believe i will put my NWParameters that contains all my TLS1.3 stuff for the using param Would i use an endpoint discovered in my browser for the with param? let nwcg = NWConnectionGroup(with: endpoint, using: parameters) If that is the case wont an NWConnectionGroup only be created when the device ID of the browsing device is larger than the discovered peer? So the device that gets connected to via its listener wouldn't have a NWConnectionGroup? Also, would I have a NWConnectionGroup for each peer im connected to? I may have a picture of how things would work... As you know i have a concept of a device controller. So if I have three devices connected to each other (not in a star configuration - more server client) there will be a single controller. Device A is the controller Device B gets controlled Device C gets controlled Device A -> Device B Device A -> Device C Device B <-XXXX-> Device C (no connection between device B and C) Since device A is the controller I will only start its NWBrowser and i will add the discovered endpoints to my Endpoints class that conforms to NWGroupDescriptor final class Endpoints: NWGroupDescriptor { var members: [NWEndpoint] init(endpoints: [NWEndpoint]) { self.members = endpoints } } Device B and C i will only start their listeners if device B wants to be come the leader then maybe a sequence of messages could passed between device B and A and this will result in Device A stopping its NWBrowser and starting its NWListener and device B stopping its NWListener and starting its NWBrowser.

Should I be using NWConnectionGroup? With all that I'm doing am i reinventing NWConnectionGroup? I looked into using it but couldnt find any examples in a peer to peer context with bonjour.

my connect function private func connect(_ connection: NWConnection, _ deviceID: DeviceID? = nil) { connection.stateUpdateHandler = { [weak self] state in guard let self = self else { return } print("Connection state changed to: \(state)") switch state { case .preparing: print("Connection preparing") case .waiting(let error): print("Connection waiting: \(error)") case .setup: print("Connection setup") case .ready: Task { @MainActor in if let deviceID = deviceID { self.isConnected[deviceID] = true } self.messagingService?.receive(from: connection) } case .failed(let error): print("Connection failed: \(error)") case .cancelled: print("Connection cancelled") Task { @MainActor in if let deviceID = deviceID { self.isConnected[deviceID] = false } } default: break } } connection.start(queue: queue) } and send function func send(_ data: Data) { let dataLength = UInt32(data.count) var content = withUnsafeBytes(of: dataLength.bigEndian) { Data($0) } content.append(data) Task { @MainActor in for connection in connections.values { guard connection.state == .ready else { print("skipping connetion that is not ready") continue } // connection.send(content: content, completion: .contentProcessed { [weak self] error in // guard let self = self else { return } connection.send(content: content, completion: .contentProcessed { error in if let error = error { print("Failed to send message: \(error)") } else { print("Message sent successfully") } }) } } }

I have the following data structures: @Published private(set) var endpoints: [DeviceID: NWEndpoint] = [:] @Published private(set) var connections: [ConnectionID: NWConnection] = [:] @Published private(set) var connectionDevice: [ConnectionID: DeviceID] = [:] @Published private(set) var deviceConnection: [DeviceID: ConnectionID] = [:] I changed my code such that the browser will first check if its deviceID is greater than the discovered device ID - if it is then it will try and connect to it. I also changed my code to save the connection in the newConnectionHandler. With all that I am experiencing some odd behavior. I have two physical devices, an iPhone and an iPad. The iPhone's deviceID is greater than the deviceID of the iPad. Due to this the iphone will create a connection to the ipad. Also due to this the ipad will not try and create a connection to the iphone. I am able to see, via printIt(), that the iphone has one item in each data structure. The iPad only has one item in endpoints (this is due to me saving it from the ipad's browser) and one item in connections (the incoming connection from the iphone) the odd behavior is as follows: right after things hit steady state - meaning that the ipad and iphone are done discovering and connection to each other - when i go to send a message from the ipad to the iphone i get the following message on the iphone logs nw_protocol_instance_add_new_flow [C18.1.1.1:1] No listener registered, cannot accept new flow quic_stream_add_new_flow [C18.1.1.1:1] [-1fe5f3879e2c580d] failed to create new stream for received stream id 1 if i then send a message from the iphone to the ipad and then try and send a message from the ipad to the iphone - it works Here is how i add an endpoint from the browser browser.browseResultsChangedHandler = { [weak self] _, changes in guard let self = self else { return } Task { @MainActor in guard let thisDeviceID = self.devicesService?.thisDevice?.id else { print("[browseResultsChangedHandler] cant get device id") return } for change in changes { switch change { case .added(let added): guard let deviceID = self.devicesService?.deviceIDFromEndpoint(added.endpoint) else { print("[browseResultsChangedHandler - .added] could not get device id") continue } guard !self.endpoints.keys.contains(deviceID) else { print("[browseResultsChangedHandler] already added device \(deviceID)") continue } guard deviceID != thisDeviceID else { print("found myself - ignoring") continue } self.addEndpoint(added.endpoint) case .removed(let removed): print("browseResultsChangedHandler: removed peer \(removed)") guard let deviceID = self.devicesService?.deviceIDFromEndpoint(removed.endpoint) else { print("[browseResultsChangedHandler - .removed] could not get deviceID") continue } guard self.endpoints.keys.contains(deviceID) else { print("[browseResultsChangedHandler] endpoint was never added \(deviceID)") continue } self.removeEndpoint(removed.endpoint) default: continue } } } } private func addEndpoint(_ endpoint: NWEndpoint) { guard !endpointExists(endpoint) else { print("endpoint already exists") return } guard let deviceID = devicesService?.deviceIDFromEndpoint(endpoint) else { print("[removeEndpoint] could not get deviceID") return } guard let thisDeviceID = devicesService?.thisDevice?.id else { print("[removeEndpoint] could not get thisDeviceID") return } if thisDeviceID > deviceID { do { let identity = importIdentityFromPKCS12() let parameters = try NWParameters(identity: identity) let connection = NWConnection(to: endpoint, using: parameters) let connectionID = getConnectionID(connection) connections[connectionID] = connection connectionDevice[connectionID] = deviceID deviceConnection[deviceID] = connectionID connect(deviceID: deviceID) } catch { print("Failed to create connection parameters: \(error)") } } endpoints[deviceID] = endpoint devicesService?.addDevice(device: DevicesService.Device(id: deviceID, name: nil, model: nil, camera: nil)) } Here is my new connection handler listener?.newConnectionHandler = { [weak self] connection in guard let self = self else { return } print("New inbound connection received from: \(connection)") Task { @MainActor in let connectionID = self.getConnectionID(connection) self.connections[connectionID] = connection self.connect(connection) } }

That’s not expected. You’re using TCP so every connection is bidirectional. I am using QUIC I want to clarify some terminology. An inbound connection is a connection from newConnectionHandler. An outbound connection is a connection I build from a discovered endpoint in browseResultsChangedHandler. Can you confirm this metal model? Assuming the above metal model is correct and quoting you In the star architecture it doesn’t matter which peer started the connection, just that there’s a single connection between each peer. In a star architecture I should only be saving one connection between peers. How I interpret this is I need a deterministic way to make sure Peer A and Peer B don't both try and connect to each other from their respective NWBrowser. In the past I had the following tactic: Get the service name (which is the peerID) of the discovered peer Compare it against mine Only create a connection and try and connect to the peer if my peer ID is greater than the discovered peer ID If my peer ID is not greater than I do nothing and assume that the discovered peer will discover me and try to connect to me For sake of this example let's assume: Device A has an ID of "a" Device B has an ID of "b" "a" < "b" == true Due to the above then Device A will discover Device B but wont try to connect to it. Device B will discover Device A and will create a connection and try to connect to it. Device B will save the connection it created to Device A. Device A will get a new connection in newConnectionHandler and will save it. Since there is no way to identify which peer the new connection came from Device B will need to send its info (device ID and device name) to Device A so it can properly map the device ID to the connection. And what im reading is that both devices should be able to use that single connection to send data back and forth. So Device A should be able to use the connection that come in newConnectionHandler to send data to Device B. Is this correct?

Thank you for the info. So me not being able to use an incoming connection (from listener.newConnectionHandler) to send data over it is expected? If I cant send data over it then do I need to keep it so i can call cancel on it if the device saving it goes offline or the app is killed? I guess since I wasn't able to use an incoming connection to send data over i thought it was useless to me. Since you're saying I should be saving the incoming connection - what do I do with it - what is the expected use of it?

No. The issue here is that client A and client B can discover and connect to each other simultaneously. You need to find a deterministic way to deduplicate such connections. I touched on this in Moving from Multipeer Connectivity to Network Framework but I’ve expanded that discussion with more specifics. Check out the newly updated Create a peer identifier section. I am still unclear if I should save the connection I get in listener?.newConnectionHandler = { [weak self] connection in Currently I have the following that works: star network (all devices connected to all devices) on each device i start a browser and a listener i save the endpoint found in the browser and i create a connection from said endpoint (i dont start the connection yet - the user needs to tap a button to start the connection to the discovered device --- this may change where i auto connect to all discovered devices) in listener?.newConnectionHandler i do setup the connection i get - i call connection.receive on it So in my case i have Device A which discovered Device B - saved Device B endpoint and created a connection. Device A has saved the endpoint of Device B and the created connection to Device B (all this happened in the NWBrowser) Device B has discovered Device A - saved Device A endpoint and created a connection. Device B has saved the end of Device A and the created connection to Device A (all this happened in the NWBrowser) I think all this is confusing because of the following test i did: i blindly saved every discovered endpoint i blindly saved every connection created from every discovered endpoint i blindly saved every connection from my listener newConnectionHandler In the end, on each device (i only had two devices), i roughly had 6 total connections in my connections array. I had everything setup to be able to send messages back and forth. When i tapped the button on Device A to send a hello world message to all the connections in the connections array Device B ever only received the hello world message twice. Two of the connections in Device A's connections array where connections I created from endpoints i discovered. The rest were connections i saved from my listener. From that I concluded that I cannot send data through connections saved from my listener?.newConnectionHandler. Am I wrong?

Thank you for that link! I read it and learned a lot. Areas I can change in my app: generate the peerID each time the app starts put the peerID in txtRecord and not in the service name Some notes about my app (which uses star architecture): In the browser i ignore my own endpoint (i see that the browser discovers its device's listener) In the browser I first check to see if i have a connection for a discovered peer - if not then i create a connection and save it - if i do already have a connection for the discovered peer then i do nothing with it I see in your post that you show code to save the new connection in the listener.newConnectionHandler. Is this in the context of a client / server? Just so I can get a handle on things: If one uses the client / server architecture - the server is the device that has a listener going and the client is the device that has the browser going. The server saves new connections in listener.newConnectionHandler and the client saves the connections it creates with the NWEndpoints it discovers. If one is using the star architecture (all devices connected to all devices) then each device has a listener going and a browser going and ONLY needs to save the connections created from the NWBrowser (ie discovered NWEndpoints). Is that right? In my app there will be basic message / command passing (ie "I'm the new leader"). I can use a basic QUIC connection for that. I also will want to transfer photos and videos. I believe I can again use a QUIC connection for this - with chunking. I want to live stream video from one device to another and for this (based on your article) the advised pattern would be opening a new UDP connection to do the live video stream? Looking at your article right now I'm seeing the section on "Start a stream". Would it be advisable to use a QUIC connection to a peer to start a new stream to then use to stream video data? From your post: If you’re using QUIC for your reliable connection, start a new QUIC stream over that connection. This is one place that QUIC shines. You can run an arbitrary number of QUIC connections over a single QUIC connection group, and QUIC manages flow control (see below) for each connection and for the group as a whole. Does that mean a stream is a QUIC connection in a QUIC Connection Group? For my use cases: simple commands / chunked video / photo files live streamed video Would I want to create a QUIC Connection Group and open two connections - one connection to handle #1 and one connection to handle #2? Thank you for your time and help.