Post

Replies

Boosts

Views

Activity

Reply to Why udp port 53 traffic is captured?
As confirmed that by only capturing all udp traffic can cause the network crash. Tried different approaches around udp, but all failed. That include exclude reserved ip segments for local network, exclude multicast segments, only include 1.0.0.0/8 to 127.0.0.0/8 for udp traffic. So far, I do believe that if there is any way to exclude udp 53 traffic, should get resolved. But as you see, I am using none zero default route as suggested, but udp 53 packets still comes in. Furthermore, as you side noted that " you cannot filter localNetwork traffic with NEAppProxyProvider either", does it mean that we have no way to eliminate udp 53 traffic with NEAppProvider? Thanks in advance for any suggestion.
May ’21
Reply to Is it possible to use AppProxy and DNSProxy together
Hi there, Following this topic, we managed to make both appproxy and dnsproxy work together at the same time, but have some constraints. Wonder if you can provide some help. Need uninstall extension each time before you run the application. It is with below command to control the extension. systemextensionsctl uninstall $DNSExtensionID $DNSExtensionName If start dnsproxy first, then only one network service is prompted and added without the one for appproxy. If start approxy first, both network service are added, but dnsproxy doesn’t work (as the extension access right problem). AppDelegate.swift:applicationDidFinishLaunching() { ...      // Start App Proxy     ProxyManager.shared().loadProxyManager(connect: true) {       ProxyManager.shared().addObserver {}       ProxyManager.shared().start() // Start the DNS Proxy DNSManager.shared().start() {}     } ... Any suggestion on this? Stability thing, mainly from dns proxy.  After running for a while with both proxies, name resolution failed. We will go on checking this point as well.
May ’21
Reply to Why udp port 53 traffic is captured?
You can let the system handle all UDP flows. You can attempt to target a small subset of UDP flows and if you receive port 53 flows, you can proxy them. Proxying UDP flows is something that can be done without a crash. That is what I am trying at the moment. But when letting all udp traffic through with filter setting like below, within around three hours, network is gone like below. Crash might not happen though. ping apple.com ... Request timeout for icmp_seq 14745 ping: sendto: No route to host Request timeout for icmp_seq 14746 ping: sendto: No route to host I have tried both udp session and NWConnection for remote connection, same result. Any suggestion? for seg in 1...254 { if seg != 127 { hosts = [("\(seg).0.0.0", "0")] for host in hosts { let ep = NWHostEndpoint(hostname: host.0, port: host.1) let rule = NENetworkRule.init(remoteNetwork: ep, remotePrefix: 8, localNetwork: nil, localPrefix: 0, protocol: .UDP, direction: .outbound) rules.append(rule) } } }
May ’21
Reply to Is it possible to use AppProxy and DNSProxy together
Hi Matt, We still cannot make AppProxyProvider and DNSProxyProvider working together. Main problems still are: Stability thing  After running for a while with both proxies, name resolution failed. Need uninstall extension each time before you run the application. It is with below command to control the extension. systemextensionsctl uninstall 5S77G864UH com.familyzone.macappproxy.dns However, we made it work by using only AppProxyProvider to filter and handle dns traffics based on udp port 53. The problem is that is this way illegal, as nowhere I can find in the document saying that is a valid way of handling dns traffic using only AppProxyProvider. So, need your help in two respects: Solve the issues having both AppProxyProvider and DNSProxyProvider working together. Help confirm whether it is legal to handle dns traffic with AppProxyProvider only? Thanks in advance. Richard
May ’21
Reply to BigSur 11.5 Beta breaks NWConnection
If I am reading this correctly, NWConnection is not broken in macOS 11.5 Beta but you are instead seeing variable network speeds through your provider where NWConnection is used. Is that correct? It is not only network speed drop, but after some time, no network access anymore (I am capturing tcp 80/443 outbound traffic only). That was not the case with bigSur 11.4. Are you using NEAppProxyProvider or are you using NETransparentProxyProvider? I am using NEAppProxyProvider.
May ’21
Reply to Is it possible to use AppProxy and DNSProxy together
Using NEAppProxyProvider or even better yet, NETransparentProxyProvider is one known alternative to creating a NEDNSProxyProvider to handle your system level DNS flows. Good to see that confirmation. As far as your name resolution issues, if you start your NEDNSProxyProvider to handle DNS flows and then you start a NEAppProxyProvider or NETransparentProxyProvider afterwards to only handle TCP flows, do you still run into a conflict here? Tried before as remember with out udp filtering from appproxy with the same problem (network cannot pass through). Can double confirm anyway. Is there any concern who to start first, dnsproxy or appproxy?
May ’21
Reply to Is it possible to use AppProxy and DNSProxy together
Hi Matt, About bringing up both dns and transparent proxy network devices, can you share with me your code on how to start them only after the other? I tried different combinations inside AppDelegate.swift, with below way, only sometimes I can get both network devices added, other times only appproxy device can be seen. func applicationDidFinishLaunching(_ aNotification: Notification) {     ProxyManager.shared().loadProxyManager(connect: true) {       ProxyManager.shared().addObserver {}       ProxyManager.shared().start()       DNSManager.shared().start() {}     }   } About stability thing with both proxy running, have you observe its stability over time? I tried with below script and found sometimes it just get stuck there for a while then resume going on. #!/bin/bash i=0 while true do   i=$((i+1))   echo "loop $i"   nslookup www.google.com   sleep 1 done
Jun ’21
Reply to BigSur 11.5 Beta breaks NWConnection
Hi Matt, Confirmed that beginning from BigSur 11.4 the network loss issue happens: I tried on two macbook pro. They have no problem when at BigSur 11.3.1. But as soon as I upgrade them to 11.4, issue happens. Already use NETransparentProxy. Can you help double confirm? You can use fast.com and speedtest.net which can show the symptom rather quick. Thanks in advance.
Jun ’21
Reply to Hundreds of appproxy flows are generated when visiting some web site.
Thanks Matt for the reply. Paste below the rules setting. speedtest.net.txt wenxuecity.com.txt private func buildRules(fullMode: Bool) -> [NENetworkRule] { var hosts = [("", "")] var rules: [NENetworkRule] = [] hosts = [("0.0.0.0", "80"), ("0.0.0.0", "443")] for host in hosts { let ep = NWHostEndpoint(hostname: host.0, port: host.1) let rule = NENetworkRule.init(remoteNetwork: ep, remotePrefix: 0, localNetwork: nil, localPrefix: 0, protocol: .TCP, direction: .outbound) rules.append(rule) } for seg in 1...223 { if seg != 127 { let ep = NWHostEndpoint(hostname: "\(seg).0.0.0", port: "0") // capture all udp traffic including port 53 var rule = NENetworkRule.init(remoteNetwork: ep, remotePrefix: 8, localNetwork: nil, localPrefix: 0, protocol: .UDP, direction: .outbound) rules.append(rule) } } return rules } private func exceptRules() -> [NENetworkRule] { var hosts = [("", "")] var rules: [NENetworkRule] = [] // rule out dhcp, ntp traffic hosts = [("0.0.0.0", "67"), ("0.0.0.0", "68"), ("0.0.0.0", "123")] for host in hosts { let ep = NWHostEndpoint(hostname: host.0, port: host.1) let rule = NENetworkRule.init(remoteNetwork: ep, remotePrefix: 0, localNetwork: nil, localPrefix: 0, protocol: .UDP, direction: .outbound) rules.append(rule) } return rules } override func startProxy(...) { ... let settings = NETransparentProxyNetworkSettings.init(tunnelRemoteAddress: "127.0.0.1") settings.includedNetworkRules = buildRules() settings.excludedNetworkRules = exceptRules() setTunnelNetworkSettings(settings) { ... } } Attached also traces with these two websites. Only printout target url when a difference flowId (flows.hash) request is up. We can see that: The number of request is significant Even with different flowId, there are many continuous request with the same url. Not sure how it is decided to generate a new flowId. Thanks in advance for the support. Richard
Jul ’21
Reply to Hundreds of appproxy flows are generated when visiting some web site.
Attach another log with flowId (tcpflow.hash) on display. You can see that when visiting speedtest.net, although some traffic with the same url, their flowId are different. Is it possible to have only one flowId generated with the same url (tcp)? [speedtest.net.txt](https://developer.apple.com/forums/content/attachment/7ab24bbc-507e-40f9-a256-e160e0727131) https://fastlane.rubiconproject.com, flowId: 416043267 https://fastlane.rubiconproject.com, flowId: 3175036210 https://fastlane.rubiconproject.com, flowId: 1261766144
Jul ’21
Reply to Why udp port 53 traffic is captured?
As confirmed that by only capturing all udp traffic can cause the network crash. Tried different approaches around udp, but all failed. That include exclude reserved ip segments for local network, exclude multicast segments, only include 1.0.0.0/8 to 127.0.0.0/8 for udp traffic. So far, I do believe that if there is any way to exclude udp 53 traffic, should get resolved. But as you see, I am using none zero default route as suggested, but udp 53 packets still comes in. Furthermore, as you side noted that " you cannot filter localNetwork traffic with NEAppProxyProvider either", does it mean that we have no way to eliminate udp 53 traffic with NEAppProvider? Thanks in advance for any suggestion.
Replies
Boosts
Views
Activity
May ’21
Reply to Is it possible to use AppProxy and DNSProxy together
Hi there, Following this topic, we managed to make both appproxy and dnsproxy work together at the same time, but have some constraints. Wonder if you can provide some help. Need uninstall extension each time before you run the application. It is with below command to control the extension. systemextensionsctl uninstall $DNSExtensionID $DNSExtensionName If start dnsproxy first, then only one network service is prompted and added without the one for appproxy. If start approxy first, both network service are added, but dnsproxy doesn’t work (as the extension access right problem). AppDelegate.swift:applicationDidFinishLaunching() { ...      // Start App Proxy     ProxyManager.shared().loadProxyManager(connect: true) {       ProxyManager.shared().addObserver {}       ProxyManager.shared().start() // Start the DNS Proxy DNSManager.shared().start() {}     } ... Any suggestion on this? Stability thing, mainly from dns proxy.  After running for a while with both proxies, name resolution failed. We will go on checking this point as well.
Replies
Boosts
Views
Activity
May ’21
Reply to Why udp port 53 traffic is captured?
You can let the system handle all UDP flows. You can attempt to target a small subset of UDP flows and if you receive port 53 flows, you can proxy them. Proxying UDP flows is something that can be done without a crash. That is what I am trying at the moment. But when letting all udp traffic through with filter setting like below, within around three hours, network is gone like below. Crash might not happen though. ping apple.com ... Request timeout for icmp_seq 14745 ping: sendto: No route to host Request timeout for icmp_seq 14746 ping: sendto: No route to host I have tried both udp session and NWConnection for remote connection, same result. Any suggestion? for seg in 1...254 { if seg != 127 { hosts = [("\(seg).0.0.0", "0")] for host in hosts { let ep = NWHostEndpoint(hostname: host.0, port: host.1) let rule = NENetworkRule.init(remoteNetwork: ep, remotePrefix: 8, localNetwork: nil, localPrefix: 0, protocol: .UDP, direction: .outbound) rules.append(rule) } } }
Replies
Boosts
Views
Activity
May ’21
Reply to Why udp port 53 traffic is captured?
Also, can you help indicate how to update in the tsi ticket? I would like to go on discussion fromTSI case 767665663, but don't know what is the link for it. Thanks in advance.
Replies
Boosts
Views
Activity
May ’21
Reply to BigSur 11.5 Beta breaks NWConnection
The version tag of current system is: 11.5 Beta (20G5023d) After some time testing, not able to browse anymore although appproxy device is still connected and can still ping through.
Replies
Boosts
Views
Activity
May ’21
Reply to Is it possible to use AppProxy and DNSProxy together
Hi Matt, We still cannot make AppProxyProvider and DNSProxyProvider working together. Main problems still are: Stability thing  After running for a while with both proxies, name resolution failed. Need uninstall extension each time before you run the application. It is with below command to control the extension. systemextensionsctl uninstall 5S77G864UH com.familyzone.macappproxy.dns However, we made it work by using only AppProxyProvider to filter and handle dns traffics based on udp port 53. The problem is that is this way illegal, as nowhere I can find in the document saying that is a valid way of handling dns traffic using only AppProxyProvider. So, need your help in two respects: Solve the issues having both AppProxyProvider and DNSProxyProvider working together. Help confirm whether it is legal to handle dns traffic with AppProxyProvider only? Thanks in advance. Richard
Replies
Boosts
Views
Activity
May ’21
Reply to BigSur 11.5 Beta breaks NWConnection
If I am reading this correctly, NWConnection is not broken in macOS 11.5 Beta but you are instead seeing variable network speeds through your provider where NWConnection is used. Is that correct? It is not only network speed drop, but after some time, no network access anymore (I am capturing tcp 80/443 outbound traffic only). That was not the case with bigSur 11.4. Are you using NEAppProxyProvider or are you using NETransparentProxyProvider? I am using NEAppProxyProvider.
Replies
Boosts
Views
Activity
May ’21
Reply to Is it possible to use AppProxy and DNSProxy together
Using NEAppProxyProvider or even better yet, NETransparentProxyProvider is one known alternative to creating a NEDNSProxyProvider to handle your system level DNS flows. Good to see that confirmation. As far as your name resolution issues, if you start your NEDNSProxyProvider to handle DNS flows and then you start a NEAppProxyProvider or NETransparentProxyProvider afterwards to only handle TCP flows, do you still run into a conflict here? Tried before as remember with out udp filtering from appproxy with the same problem (network cannot pass through). Can double confirm anyway. Is there any concern who to start first, dnsproxy or appproxy?
Replies
Boosts
Views
Activity
May ’21
Reply to Is it possible to use AppProxy and DNSProxy together
Hi Matt, About bringing up both dns and transparent proxy network devices, can you share with me your code on how to start them only after the other? I tried different combinations inside AppDelegate.swift, with below way, only sometimes I can get both network devices added, other times only appproxy device can be seen. func applicationDidFinishLaunching(_ aNotification: Notification) {     ProxyManager.shared().loadProxyManager(connect: true) {       ProxyManager.shared().addObserver {}       ProxyManager.shared().start()       DNSManager.shared().start() {}     }   } About stability thing with both proxy running, have you observe its stability over time? I tried with below script and found sometimes it just get stuck there for a while then resume going on. #!/bin/bash i=0 while true do   i=$((i+1))   echo "loop $i"   nslookup www.google.com   sleep 1 done
Replies
Boosts
Views
Activity
Jun ’21
Reply to BigSur 11.5 Beta breaks NWConnection
Hi Matt, Confirmed that beginning from BigSur 11.4 the network loss issue happens: I tried on two macbook pro. They have no problem when at BigSur 11.3.1. But as soon as I upgrade them to 11.4, issue happens. Already use NETransparentProxy. Can you help double confirm? You can use fast.com and speedtest.net which can show the symptom rather quick. Thanks in advance.
Replies
Boosts
Views
Activity
Jun ’21
Reply to BigSur 11.5 Beta breaks NWConnection
And filter out only tcp 80 and 443 as I do.
Replies
Boosts
Views
Activity
Jun ’21
Reply to Is it mandatory to place appproxy-sytemextension app directly under /Applications folder
Thanks a lot Matt for the. reply. That's clear.
Replies
Boosts
Views
Activity
Jul ’21
Reply to Hundreds of appproxy flows are generated when visiting some web site.
Thanks Matt for the reply. Paste below the rules setting. speedtest.net.txt wenxuecity.com.txt private func buildRules(fullMode: Bool) -> [NENetworkRule] { var hosts = [("", "")] var rules: [NENetworkRule] = [] hosts = [("0.0.0.0", "80"), ("0.0.0.0", "443")] for host in hosts { let ep = NWHostEndpoint(hostname: host.0, port: host.1) let rule = NENetworkRule.init(remoteNetwork: ep, remotePrefix: 0, localNetwork: nil, localPrefix: 0, protocol: .TCP, direction: .outbound) rules.append(rule) } for seg in 1...223 { if seg != 127 { let ep = NWHostEndpoint(hostname: "\(seg).0.0.0", port: "0") // capture all udp traffic including port 53 var rule = NENetworkRule.init(remoteNetwork: ep, remotePrefix: 8, localNetwork: nil, localPrefix: 0, protocol: .UDP, direction: .outbound) rules.append(rule) } } return rules } private func exceptRules() -> [NENetworkRule] { var hosts = [("", "")] var rules: [NENetworkRule] = [] // rule out dhcp, ntp traffic hosts = [("0.0.0.0", "67"), ("0.0.0.0", "68"), ("0.0.0.0", "123")] for host in hosts { let ep = NWHostEndpoint(hostname: host.0, port: host.1) let rule = NENetworkRule.init(remoteNetwork: ep, remotePrefix: 0, localNetwork: nil, localPrefix: 0, protocol: .UDP, direction: .outbound) rules.append(rule) } return rules } override func startProxy(...) { ... let settings = NETransparentProxyNetworkSettings.init(tunnelRemoteAddress: "127.0.0.1") settings.includedNetworkRules = buildRules() settings.excludedNetworkRules = exceptRules() setTunnelNetworkSettings(settings) { ... } } Attached also traces with these two websites. Only printout target url when a difference flowId (flows.hash) request is up. We can see that: The number of request is significant Even with different flowId, there are many continuous request with the same url. Not sure how it is decided to generate a new flowId. Thanks in advance for the support. Richard
Replies
Boosts
Views
Activity
Jul ’21
Reply to Hundreds of appproxy flows are generated when visiting some web site.
Attach another log with flowId (tcpflow.hash) on display. You can see that when visiting speedtest.net, although some traffic with the same url, their flowId are different. Is it possible to have only one flowId generated with the same url (tcp)? [speedtest.net.txt](https://developer.apple.com/forums/content/attachment/7ab24bbc-507e-40f9-a256-e160e0727131) https://fastlane.rubiconproject.com, flowId: 416043267 https://fastlane.rubiconproject.com, flowId: 3175036210 https://fastlane.rubiconproject.com, flowId: 1261766144
Replies
Boosts
Views
Activity
Jul ’21
Reply to Hundreds of appproxy flows are generated when visiting some web site.
And the question also is: with a new flowId request in, we will create a new tcp connection (so that is quite a lot!). Is that correct?
Replies
Boosts
Views
Activity
Jul ’21