Post

Replies

Boosts

Views

Activity

Reply to I want to use a QUIC stream with Swift's NWProtocolQUIC
@takt so I was running into something similar and I am still chasing it done but I have some notes worth sharing: While you mentioned it's probably not the server side I have seen an issue (or think so) where the NWConnectionGroup is establishing an initial stream. When the connection is established to the server , you are generating a new stream and sending data which will, on the server side, trigger the stream to be "opened". What I found was if your server side code is something like this (sudo code for readability): conn = endpoint.accept_connection() stream = conn.accept_new_stream() ... let bytes = stream.recv() You might find that your server is accepting a stream but you don't get the bytes. AFAICT, this is because there is an initial stream being establish silently by the NWConnectionGroup which you cannot get a handle to. When you create your new stream on your client, and send data this causes the hidden stream to be opened as well per the QUIC spec: A stream ID that is used out of order results in all streams of that type with lower-numbered stream IDs also being opened. https://www.rfc-editor.org/rfc/rfc9000.html#name-stream-types-and-identifier This means you have a handle to the initial hidden stream and the server will accept data from your newly open stream. From my testing you need to make sure that your server is accepting streams multiple times: conn = endpoint.accept_connection() _ = conn.accept_new_stream() stream = conn.accept_new_stream() ... and then I found I want able to process the incoming data. Reference this example repo and issue where I dug into this will the Rust implementation of QUIC and Network.framework https://github.com/paxsonsa/quic-swift-demo https://github.com/quinn-rs/quinn/issues/1742 Cheers!
Jan ’24
Reply to Weird behaviour with Network Framework and QUIC Multiplex Groups.
I understand! Thanks! To summarize: No, the options for a connection group for a given endpoint cannot differ. If you want to mix stream directionality options cannot do so with a NWConnectionGroup as there is a constrain that for a given endpoint, new "stream" cannot redefine the options of the main group. What happens is that new streams are derived from the initial QUIC connection that were used to setup the tunnel, so typically those new streams would take on the initial setup options of the main tunnel (or groups) connection. And this makes sense because otherwise you would have streams that would have different connection parameters (and behavior) than the parent connection / tunnel. And if thats happens then a case could be made for an entirely new connection or tunnel. I see, that make sense from an NW API pov and the way it works under the hood, even if that is not a limitation of the QUIC protocol itself. It would be nice if we could change could established a new stream with different directionality as that is not a property of the connection socket itself but of the streams which is legal for QUIC. By establishing a new connection that will give me the ability to change stream directionality BUT from a server perspective that will be a completely new accepted connection and that's something servers need to keep in mind that may organize operations around a single connection. It's doable about its an unfortunate limitation (feature?) of the well NWConnectionGroups with NWMultiplexGroup parameters work. I would seem to me that the the directionality of streams should not be a property of the connection group as a whole and something that can be overridden for the new streams but I am no QUIC expert :)
Jan ’24
Reply to Weird behaviour with Network Framework and QUIC Multiplex Groups.
The second issue of the hidden stream is more complex and requires some understanding of the server. Effectively, the server does something like this (sudo code, original code in the repo provided): let conn = endpoint.accept_connection() print( "[server] connection accepted: addr=\(conn.remote_address())", ) print("[server] waiting for bi stream"); let (send_end, recv_end) = conn.accept_bi() print("[server] awaiting recv") recv_end.recv(....) Now if I establish a connection with options.direction = .bidirectional on the client side: let options = NWProtocolQUIC.Options(alpn: ["demo"]) // Set the initial stream direction. options.direction = .bidirectional ... let parameters = NWParameters(quic: options) let descriptor = NWMultiplexGroup(to: endpoint) let group = NWConnectionGroup(with: descriptor, using: parameters) and then in the group state handler, once the connection is ready I make a new stream (same directionality): group.stateUpdateHandler = { newState in switch newState { case .ready: print("Connected using QUIC!") let options = NWProtocolQUIC.Options(alpn: ["demo"]) options.direction = .bidirectional mainConn = group.extract()! // force unwrap print("new stream made: \(mainConn)") ... case .failed(let error): print("main connection failed.") break } } ... When sending bytes through that new connection from the swift client, on the server side I will get logs like so: [server] connection accepted: addr=127.0.0.1:53890 [server] waiting for bi stream [server] awaiting recv The server will receive the connection, wait for the bidirectional stream to be opened and then await the bytes but the bytes will never come through the newly accepted stream. I can see the swift client sending bytes and in higher verbosity levels the server IS getting the bytes. quinn_proto::connection: got Data packet (67 bytes) from 127.0.0.1:56022 using id 8c3bc319a7211707 quinn_proto::connection: got Data packet (67 bytes) from 127.0.0.1:56022 using id 8c3bc319a7211707 but the only way for me start to recving the bytes from the stream accepted on the server and opened on the swift client in the group state handler is to accept the bidirectional stream twice. print("[server] waiting for bi stream"); let (_, _) = conn.accept_bi() let (send_end, recv_end) = conn.accept_bi() I have already investigated with the maintainers of the package I use that is not a weird bug on their end and in higher verbosity levels we can see TWO stream being opened. But on the swift side, as far as I can see we are only establish a single stream and that 'first' stream is not accessible from the NWConnectionGroup at all. If I don't send bytes that the server never makes to even attempting to recv bytes but I do think this indicates that there is stream being initiated under the hood of the NWConnection and the RFC quit does outline this behavior: [A stream ID that is used out of order results in all streams of that type with lower-numbered stream IDs also being opened. ](https://www.rfc-editor.org/rfc/rfc9000.html#name-stream-types-and-identifier) Is there a way for me to increase logging or inspect further where this initial stream is being opened?
Jan ’24
Reply to Weird behaviour with Network Framework and QUIC Multiplex Groups.
@meaton thanks for having a look and the example! I think there was a misunderstanding but I realize my code was missing a piece! I am not struggling with the connection aspect or the error you mentioned. I am conflating to two issue, apologies! Creating a new stream with new options for the stream. I noticed that there appears to be a hidden stream created when we create the connection with the NWConnectionGroup which is the more complex issue. If my initial connection is established with options specified like so: let options = NWProtocolQUIC.Options(alpn: ["demo"]) // Set the initial stream to unidirectional. options.direction = .unidirectional ... let parameters = NWParameters(quic: options) let descriptor = NWMultiplexGroup(to: endpoint) let group = NWConnectionGroup(with: descriptor, using: parameters) and after establishing a connection and the group set is .ready I am trying something like. group.stateUpdateHandler = { newState in switch newState { case .ready: print("Connected using QUIC!") let options = NWProtocolQUIC.Options(alpn: ["demo"]) options.direction = .bidirectional mainConn = group.extract(connectionTo: endpoint, using: options)! // force unwrap print("new stream made: \(mainConn)") ... case .failed(let error): print("main connection failed.") break } } ... I am seeing this this error in the console log and specifically the nw_endpoint_flow_failed_with_error and then the new stream connection is in a failed state. Connected using QUIC! new stream made: Optional([C3 127.0.0.1:4567 quic, attribution: developer, attach protocol listener]) nw_endpoint_flow_setup_cloned_protocols [C3 127.0.0.1:4567 in_progress socket-flow (satisfied (Path is satisfied), viable, interface: lo0)] could not find protocol to join in existing protocol stack nw_endpoint_flow_failed_with_error [C3 127.0.0.1:4567 in_progress socket-flow (satisfied (Path is satisfied), viable, interface: lo0)] failed to clone from flow, moving directly to failed state Main Connection State: failed(POSIXErrorCode(rawValue: 50): Network is down) main connection failed. quic_recovery_pto PTO fired after validation If I remove the endpoint and options from the extract call: mainConn = group.extract(connectionTo: endpoint, using: options)! // force unwrap // into mainConn = group.extract()! // force unwrap If works but my streams is not bidirectional. Am I able to create new streams with different directions in this API? Or do I have to choice between bidirectional or unidirectional for the entire group?
Jan ’24
Reply to Weird behaviour with Network Framework and QUIC Multiplex Groups.
Additionally, there seems to be some weird issues with a hidden stream being created but there is no way to gain access to the handle: Hidden Streams My server is simple, it accepts a single connect and then a single stream initiated by the client. In my swift code I am looking to establish a connection and create a bidirectional stream after the multiplex group is ready. After the group is ready I extract a new stream from the group and attempt to send data once it is ready. However, this new stream is never accepted but on the server side, the hidden stream is accepted. Upon a deeper dive into the logging, I can see that there are two stream open by the end of the program, the newest one is the one I created by extracting it from the group, the first is one I did not create and do not have a handle too. If I change my server to accept twice, the stream I create after the group is ready is usable. This is odd behaviour, I don't have a handle to the first hidden stream on the client side and attempt to send data with the group handle just creates new streams.
Jan ’24