I get the same results with both overridden methods -- many things have a local port of 0, but some have been bound. eg 2023-03-20 09:39:59.799074+0000 0x16431f5 Debug 0x0 71647 0 com.kithrup.NETest.NETestTunnel: [com.kithrup:NETestTuennel] Got UDP Flow UDP io.tailscale.ipn.macos.network-extension[{length = 20, bytes = 0x9aa36e7f0ce1e8c94d16d312b51d6b8820b52ff4}] local port 52711 interface en0(bound) remoteHostName <none> localEndpoint 192.168.1.58:52711

Interestingly, I don't have that method -- both TCP and UDP flows come via handleNewFlow(_:) (since that's the one the documentation says to override; I'll try overriding that one and see what happens). The local endpoint is, as far as I can tell, always filled out with the hostname being the interface's IP address, and port being 0. I wrote a small program to open two UDP sockets and connect one, and that did not provide any difference in behaviour that I could see -- I did that before asking. 😄

The responses do not have to come from the destinations. Consider, if you will, a UDP-based bit-torrent service: you send a datagram to one particular node, saying "gimme this file." That node then broadcasts that around. You then start getting datagrams from thousands of different nodes, none of which you'd actually sent a message to in the first place. That is only possible if the application did not use connect(). If, however, the application did use connect()... then it should only get responses from that particular node. How is the transparent proxy supposed to be able to tell what the application is expecting?

Ah ha! Ok, that makes sense.

Oh, this doe seem to have been ARC-related, although I'm not sure why it worked before. I changed my cached ivar to an @property with retain, and now it's back to not crashing, so yay!

Ok, this is totally bizarre: I just aded     os_log_debug(self.logger, "returning cached connection %p", _connection); to the method that handles the cached version, and now it's not crashing. Comment it out, and crash. ARC related?

Sorry I wasn't clear enough there -- I'm talking about when the application is doing read or recvfrom or other on a UDP socket. Most applications that use UDP, as I said, will either use a timeout of some sort, or a non-blocking socket. (And, of course, they may also be using kqueue to find the same information out.) So how should a Transparent Proxy Provider, which does not know if the socket was set to be non-blocking, or has a timeout, deal with a UDP flow? Or should the TPP instead always assume that if the write-to-internet side (the readDatagrams method) gets closed, the read-from-internet side is also closed?

And solved -- it needs to have -target ${arch}-apple-${os}${version}, eg x86_64-apple-macos11.5. And yes, it was vcpkg doing this. I can fix it by editing the triplets file, or I might some day do up a better diff and see how that goes. Or we could just drop support of macOS 11. But options, yay!

Ok. So: if I use Xcode to build (and set the Deployment Target to 11.5), then the resulting binary works. We use vcpkg to manage a bunch of 3rd party libraries, and I'm going to guess that's the cause there -- it doesn't build with xcodebuild, and doesn't have a .xcodeproj bundle. I'm not sure what xcodebuild is doing differently in its invocations, but I could try to go down that rabbithole. 😄

So normally if you're a process doing UDP I/O, you use a timeout of some sort (usually with recvfrom, or a read with an alarm signal or something). How is a network extension supposed to know that? Or is it supposed to assume that if a process signals done-with-writing, that it should treat both directions as closed?

Yes, using std::ostreamstring by itself does the trick. And again, this isn't making any sense to me. I compiled up the same program (using c++ -g t.cc -o macos<whatever>) and did: nm macos11 | grep ostreamIc | sort | grep U ; echo '****' ; nm macos12 | grep ostreamIc | sort | grep U                  U __ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_                  U __ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev                  U __ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEED0Ev                  U __ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEED1Ev                  U __ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEED2Ev                  U __ZTINSt3__113basic_ostreamIcNS_11char_traitsIcEEEE                  U __ZTv0_n24_NSt3__113basic_ostreamIcNS_11char_traitsIcEEED0Ev                  U __ZTv0_n24_NSt3__113basic_ostreamIcNS_11char_traitsIcEEED1Ev ****                  U __ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_                  U __ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev                  U __ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEED2Ev which to me looks like macos12 has fewer undefineds, but the ones it does have are in the same set as macos11. Is this a bug?

And on macOS 12: On macOS 12: Apple clang version 14.0.0 (clang-1400.0.29.202) Target: x86_64-apple-darwin21.6.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx12.0.0 -Wundef-prefix=TARGET_OS_ -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name t.cc -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-strict-return -fno-rounding-math -funwind-tables=2 -target-sdk-version=13.1 -fvisibility-inlines-hidden-static-local-var -target-cpu penryn -tune-cpu generic -debug-info-kind=standalone -dwarf-version=4 -debugger-tuning=lldb -target-linker-version 820.1 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/14.0.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/usr/local/include -stdlib=libc++ -internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1 -internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/14.0.0/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-elaborated-enum-base -Wno-reserved-identifier -Wno-gnu-folding-constant -Wno-cast-function-type -Wno-bitwise-instead-of-logical -fdeprecated-macro -fdebug-compilation-dir=/Users/sef/Downloads/ExtensionPermission_7160_Crashes -ferror-limit 19 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fno-cxx-modules -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcommon -clang-vendor-feature=+messageToSelfInClassMethodIdReturnType -clang-vendor-feature=+disableInferNewAvailabilityFromInit -clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation -fno-odr-hash-protocols -clang-vendor-feature=+enableAggressiveVLAFolding -clang-vendor-feature=+revert09abecef7bbf -clang-vendor-feature=+thisNoAlignAttr -clang-vendor-feature=+thisNoNullAttr -mllvm -disable-aligned-alloc-awareness=1 -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /var/folders/x4/bk1vl6m88xjfxq006s6sv2s00000gn/T/t-8daa50.o -x c++ /tmp/t.cc clang -cc1 version 14.0.0 (clang-1400.0.29.202) default target x86_64-apple-darwin21.6.0 ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include" ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/Library/Frameworks" #include "..." search starts here: #include <...> search starts here: /usr/local/include /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/14.0.0/include /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory) End of search list. "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -no_deduplicate -dynamic -arch x86_64 -platform_version macos 12.0.0 13.1 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -o /tmp/t -L/usr/local/lib /var/folders/x4/bk1vl6m88xjfxq006s6sv2s00000gn/T/t-8daa50.o -lc++ -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/14.0.0/lib/darwin/libclang_rt.osx.a "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/dsymutil" -o /tmp/t.dSYM /tmp/t

On macOS 11: Apple clang version 12.0.5 (clang-1205.0.22.11) Target: x86_64-apple-darwin20.5.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin  "/Library/Developer/CommandLineTools/usr/bin/clang" -cc1 -triple x86_64-apple-macosx11.0.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name t.cc -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-strict-return -fno-rounding-math -munwind-tables -target-sdk-version=12.1 -fvisibility-inlines-hidden-static-local-var -target-cpu penryn -debugger-tuning=lldb -target-linker-version 650.9 -v -resource-dir /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -I/usr/local/include -stdlib=libc++ -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/include/c++/v1 -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/local/include -internal-isystem /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/include -internal-externc-isystem /Library/Developer/CommandLineTools/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-elaborated-enum-base -fdeprecated-macro -fdebug-compilation-dir /Users/sef -ferror-limit 19 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcommon -clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation -fno-odr-hash-protocols -mllvm -disable-aligned-alloc-awareness=1 -o /var/folders/m6/s94tzxq53p568j_kqqxyk3d40000gn/T/t-78ad5e.o -x c++ /tmp/t.cc clang -cc1 version 12.0.5 (clang-1205.0.22.11) default target x86_64-apple-darwin20.5.0 ignoring nonexistent directory "/usr/local/include" ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/local/include" ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/Library/Frameworks" #include "..." search starts here: #include <...> search starts here:  /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/include/c++/v1  /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/include  /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/include  /Library/Developer/CommandLineTools/usr/include  /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/System/Library/Frameworks (framework directory) End of search list.  "/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -no_deduplicate -dynamic -arch x86_64 -platform_version macos 11.0.0 12.1 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -o a.out -L/usr/local/lib /var/folders/m6/s94tzxq53p568j_kqqxyk3d40000gn/T/t-78ad5e.o -lc++ -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/lib/darwin/libclang_rt.osx.a

Oh, that'll help me find it, I think.

My sample program? c++ -g t.cc -o t Our application is built using cmake, which creates a .xcodeproj and runs xcodebuild. As I said (I think?), I'm not sure what particular bit of our code is triggering it. We've tried using -isysroot and setting the deployment target to 11.5, but it's still using /usr/lib/libc++.1.dylib and, well, see first comment about that.