We have tried doing this, to no effect. The Order property seems to be ignored when using MDM. (When we manually installed one extension, waited for it to be activated, and then the other, then flows were processed in that order.) We're using roughly this: <key>TransparentProxy</key> <dict> <key>AuthenticationMethod</key> <string>Password</string> <key>ProviderBundleIdentifier</key> <string>com.kithrup.Multiple-TPP.TPP-2</string> <key>ProviderDesignatedRequirement</key> <string>stuff</string> <key>Order</key> <integer>1</integer> <key>RemoteAddress</key> <string>localhost</string> </dict> to set that one to be the first. We have another one, different company, that we had Order set to 2. And yet, that one always got flows first, no matter what the Order value was. Are we doing something very wrong? Or is this a bug somewhere?

This is happening increasingly often, and just happened on a automated testing VM. I filed FB20555301

I worked in CoreOS for a while, I'm good with spindump files. 😄 It's almost 8pm here, so i'll look at it tomorrow.

There were .diag files in the sysdiagnose, which showed the crash message. The other one I was looking at, which was closer to 100k, had been alive for 40 seconds, based on top.txt.

there any part/component in your app where you have an unrestricted retry loop (try/fail/try... as fast as you can) involving XPC? Hm, not really. We will retry activating the extension, but that's not us using XPC directly; in another case, we'll try talking with the extension using XPC, and if it times out, we deactivate and reactivate the extension. (But that takes 2-3 minutes before it'll get that far.)

Just woke up, so truncated responses: No, not doing any Mach IPC directly -- everything is done via ObjC XPC objects in the GUI app (and Swift ones in the extension). I did notice something else happening, which is that the attempt to load/store the VPN configuration is failing with "wrong type of configuration" or something like that. This causes our GUI app to exit, in the hope that this might cause it to work the 2nd time around, although that doesn't seem to be the case. (So, basically, the GUI app is running for about 40 seconds, and then dying.)

It is signed with a Developer ID. But, as I said, my coworkers ripped a subset out of a bundle, so that presumably broke all the signing.

Wait, no, I've now been told they grabbed java from AWS, and then stripped out parts of it to make it smaller, and as a result lost the bundle structure.

This happened again, even with my attempts to work around it. I updated the fb, and attached the sysdiagnose that was generated. From looking at the logs, it looks like (to me) that it copied the extension bundle into /Library/SystemExtensions, but then decided that it had been modified? Which doesn't make sense.

This is still happening (or seeming to happen) on macOS 14.1 and 15.3.1, sigh (Oops, forgot to mention: one fun case we seem to have is: if the dialogue does not appear, but the system thinks it has, then it will never show up until the system is rebooted. An invisible dialogue box.)

Of course, that doesn't change the fact that codesign has no useful diagnostics, and does a horrible job of cleaning up after itself.... 😄

meh, it happens. But it doesn't bode well for me -- since that likely means nobody else has run into the problem. siiiiiiiiiiiiiiiiiiigh

This is the line I was adding to /etc/pf.conf on every reboot: block drop from any to 2620:149:981:603::10 ETA: I want to be clear that the ridiculous part is that it's been going on for over a year, that I never got any response even after I mentioned in at least one forum comment that it was still occurring here, and that codesign after decades continues to give no error messages on failure. Oh, also that it doesn't clean up the .cstemp files it leaves behind, which admittedly were the only clue I had what was going on.

I am still digesting that, but I was about to upload another sysdiagnose -- this one from a githubs action VM that demonstrated the same behaviour (but which was a clean install of our app). But I think I'll try to fix some of the obvious-fixable issues there. We don't have UF_IMMUTABLE set on anything, and the one process in the suite that uses ESF doesn't protect anything in /Library/SystemExtensions. That process needs the TCC, but without MDM, it requires manual intervention by the user. I don't think it does it on the github actions tests. Each build gets a new number; for annoying reasons, the build is done twice (Apple Silicon and Intel), lipo'd together, and then codesigned again. The crashes you note are either segfaults or reference count crashes, and should not happen -- it seems to be an issue with XPC. The code in question is written in ObjC.

My Friend the System Log was not at all useful: 2025-06-18 12:43:44.553820+0000 0x6de3 Default 0x0 5681 0 dsa: System extension request com.kithrup.dsa.Extension (0x60000176c8a0) failed with error The operation couldn't be completed. (OSSystemExtensionErrorDomain error 1.) 2025-06-18 12:43:44.554109+0000 0x6de3 Error 0x0 5681 0 dsa: [com.kithrup:ExtensionLoader] Request to load extension com.kithrup.dsa.Extension failed with unknown error, trying again 2025-06-18 12:43:44.554480+0000 0x6de3 Default 0x0 5681 0 dsa: System extension request com.kithrup.dsa.Extension (0x60000176bcf0) failed with error The operation couldn't be completed. (OSSystemExtensionErrorDomain error 1.) 2025-06-18 12:43:44.554635+0000 0x6de3 Error 0x0 5681 0 dsa: [com.kithrup:ExtensionLoader] Request to load/unload extension com.kithrup.dsa.Extension failed with error The operation couldn't be completed. (OSSystemExtensionErrorDomain error 1.) hm, a bit before that, it looks like sysextd crashed, I presume while trying to load it. It, too, is lacking in any useful information. The sysdiagnose has a crash log for sysexted, which is equally helpful: Thread 2 Crashed:: Dispatch queue: sysextd.extension_manager 0 sysextd 0x1013c3a1e 0x101369000 + 371230 1 sysextd 0x1013abc0b 0x101369000 + 273419 2 sysextd 0x1013ab8c6 0x101369000 + 272582 3 sysextd 0x1013af069 0x101369000 + 286825 4 sysextd 0x1013ab680 0x101369000 + 272000 5 sysextd 0x1013ab48c 0x101369000 + 271500 6 sysextd 0x1013ab703 0x101369000 + 272131 7 Foundation 0x7ff8057d7525 __NSXPCCONNECTION_IS_CALLING_OUT_TO_EXPORTED_OBJECT_S1__ + 10 8 Foundation 0x7ff805dad25f -[NSXPCConnection _decodeAndInvokeMessageWithEvent:reply:flags:] + 2318 9 Foundation 0x7ff805dae9d8 message_handler_message + 79 10 Foundation 0x7ff805dae4f5 message_handler + 140 11 libxpc.dylib 0x7ff80455f998 _xpc_connection_call_event_handler + 56 12 libxpc.dylib 0x7ff80455e74c _xpc_connection_mach_event + 1399 13 libdispatch.dylib 0x7ff8046760cd _dispatch_client_callout4 + 9 14 libdispatch.dylib 0x7ff8046901a7 _dispatch_mach_msg_invoke + 455 15 libdispatch.dylib 0x7ff80467c088 _dispatch_lane_serial_drain + 393 16 libdispatch.dylib 0x7ff804690cd4 _dispatch_mach_invoke + 484 17 libdispatch.dylib 0x7ff80467c088 _dispatch_lane_serial_drain + 393 18 libdispatch.dylib 0x7ff80467cd39 _dispatch_lane_invoke + 366 19 libdispatch.dylib 0x7ff8046873fc _dispatch_workloop_worker_thread + 765 20 libsystem_pthread.dylib 0x7ff804813c55 _pthread_wqthread + 327 21 libsystem_pthread.dylib 0x7ff804812bbf start_wqthread + 15 (SIGILL aka signal 4, which I vaguely recall is signing related in xnu?)