Thank you Eskimo, It’s complicated, but a key factor is your app’s designated requirement. See TN3127 Inside Code Signing: Requirements. Do I then understand correctly that TCC uses application's designate requirements for setting up the initial access.csreq field? If I change the field will it just evaluate the requirements against the application or it also compares the access.csreq with actual the DR of the application (i.e. they match)? This doesn’t make sense to me. Ad hoc signed code can’t use an App ID because an App ID must be authorised by a provisioning profile and a profile can only authorise code signed with signing identity whose certificate was issued by Apple. See TN3125 Inside Code Signing: Provisioning Profiles. I'm probably using wrong term here - I should rather use a "bundle identifier". Our build environment produces macOS application bundles, which have the same bundle identifier but one is ad-hoc signed (codesign -s - ...) with restricted entitlements stripped, which is used on PR pipelines[1]. The other one is properly signed with DeveloperID distribution certificate. Problem is that machines which run the UI tests are fetching from both of these queues and even when we split them, the churn there (install/run/uninstall/repeat) seems to break a TCC a lot. [1] This is weird but we have a few reasons for that where some are: Our security policy is that we don't sign with production distribution certificates anything which was not reviewed and merged to master . Using different bundle identifier is also not possible because many of the third-part integrations which we're also testing depend on the exact bundle id. Running code unsigned on intel is also not an option - we still need to stick entitlements on executables so we don't run into problems with sandbox later. Having provisioning profile and shared signing identity for the PR builds has a high maintenance cost because we would need to constantly update the provisioning profile with a new machine uuids (we're not small company, we're roughly of size of yours :))

Cool. Then assigning someone to automate this shouldn’t be a problem, right? (-: Haha :-D I'll share what we did in case someone trips over the same thing. I did took and advantage of the fact that our test hosts are deployed with full disk access allowed for Terminal application (which runs the test runner) so I modify user's TCC.db to allow System Settings automation and then in the test script when I need screen recording permission I run automation[1] script which adds the application under test to screen recording allowlist. Not happy with that, I would rather run sql insert into TCC.db in the fraction of the time but it works somehow reliably and we don't need to disable SIP. [1] The screen recording permissions are stored in system-wide /Library/Application Support/com.apple.TCC which has rootless attribute and can't be modified directly without disabling SIP

Thank you @eskimo for quick answer :-) Please post your bug number, just for the record. Here it is FB13136758. If removing these entitlements helps with this problem — and I’m not 100% sure it will (...) It does help because I can then explicitly add com.apple.security.get-task-allow which will otherwise prevent the helper from starting if com.apple.security.inherit is present. Not sure if this is bug or feature :-) The docs are clear that any other entitlement than those two for sandbox inheritance will abort the application start but that is not entirely true already. For anyone having the same problem: Solution for me wast to remove the com.apple.security.app-sandbox and com.apple.security.inherit and add com.apple.security.get-task-allow so I can debug the helper which will still be running in parent's sandbox. For production builds I reverse that and it should work for Mac App Store too hopefully without any unexpected side-effects.

I think it’s reasonable to argue that com.apple.security.get-task-allow should be allowed to facilitate debugging. If you agree, make your case in a bug report. Agree, suggested that in case FB13163106, hopefully someone finds it out worth of implementation :)

Thank you @eskimo I was digging more and I found the source of problem and repro[1] - in the dynamic library we load some static intializer is creating a nw_path_monitor and once we fork the process it crashes in the atfork handler when Network frameworks tries to cleanup. I'll report a bug and see if someone tells me if it's operator error or just a bug :-) I would still have a one more question for educational purposes: Are the the posix_spawn and NSTask doing something fundamentally different to just calling fork and exec*? I mean calling a mach APis or other dark magic? [1] Simple main.m (used application template from Xcode) // main.m #import <Cocoa/Cocoa.h> #import <Network/Network.h> #import <dispatch/dispatch.h> int main(int argc, const char * argv[]) { @autoreleasepool { nw_path_monitor_t mon = nw_path_monitor_create(); nw_path_monitor_set_update_handler(mon, ^(nw_path_t path) { NSLog(@"monitor updated"); }); nw_path_monitor_start(mon); } pid_t pid = fork(); if (pid == -1) { NSLog(@"Fork failed"); exit(1); } if (pid == 0) { while (true) { NSLog(@"Forked child here"); sleep(1); } return 0; } return NSApplicationMain(argc, argv); } Child crashes with: Process: NWForkCrash [69516] Path: /Users/USER/Library/Developer/Xcode/DerivedData/NWForkCrash-dnvdeuuhbnuhxublasbhfmmluzqb/Build/Products/Debug/NWForkCrash.app/Contents/MacOS/NWForkCrash Identifier: com.****.NWForkCrash Version: 1.0 (1) Code Type: ARM-64 (Native) Parent Process: NWForkCrash [69508] Responsible: NWForkCrash [69508] User ID: 501 Date/Time: 2023-09-15 16:52:56.2815 +0200 OS Version: macOS 13.5.2 (22G91) Report Version: 12 Anonymous UUID: D6E5A34D-2127-16AF-16E7-BDA9139A6A82 Sleep/Wake UUID: DF75986B-D513-4000-993D-69A7AB7261A1 Time Awake Since Boot: 88000 seconds Time Since Wake: 740 seconds System Integrity Protection: enabled Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_BREAKPOINT (SIGTRAP) Exception Codes: 0x0000000000000001, 0x0000000194551238 Termination Reason: Namespace SIGNAL, Code 5 Trace/BPT trap: 5 Terminating Process: exc handler [69516] Application Specific Information: BUG IN CLIENT OF LIBPLATFORM: os_unfair_lock is corrupt Abort Cause 258 crashed on child side of fork pre-exec Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 libsystem_platform.dylib 0x194551238 _os_unfair_lock_corruption_abort + 88 1 libsystem_platform.dylib 0x19454c788 _os_unfair_lock_lock_slow + 332 2 Network 0x19b1b4af0 nw_path_shared_necp_fd + 124 3 Network 0x19b1b4698 -[NWConcrete_nw_path_evaluator dealloc] + 72 4 Network 0x19af9d970 __nw_dictionary_dispose_block_invoke + 32 5 libxpc.dylib 0x194260210 _xpc_dictionary_apply_apply + 68 6 libxpc.dylib 0x19425c9a0 _xpc_dictionary_apply_node_f + 156 7 libxpc.dylib 0x1942600e8 xpc_dictionary_apply + 136 8 Network 0x19acd5210 -[OS_nw_dictionary dealloc] + 112 9 Network 0x19b1beb08 nw_path_release_globals + 120 10 Network 0x19b3d4fa0 nw_settings_child_has_forked() + 312 11 libsystem_pthread.dylib 0x10463f7c8 _pthread_atfork_child_handlers + 76 12 libsystem_c.dylib 0x1943d9944 fork + 112 13 NWForkCrash 0x1045db024 main + 96 (main.m:21) 14 dyld 0x1941c7f28 start + 2236

Also have to add that from experimentation I learned that basically once sandbox_init[1][2] is called in the parent process, a child can't init its own sandbox, any attempt to re-init will fail. [1] Doesn't matter if triggered by entitlements via libsecinit or manually calling sandbox_init in the main() [2] The role of com.apple.security.inherit was for long time lost on me, it seems its sole role is to prevent libsecinit to init a sandbox when com.apple.security.app-sandbox is present :)

[1] Here's the full error stack: connection [0x13a904080] activating connection: mach=false listener=false peer=true name=com.example.app.service.peer[46207].0x13a904080 com.apple.xpc security_exception MacOS error: -67065 com.apple.securityd security_exception 0 Security 0x00000001943d6108 Security::CommonError::LogBacktrace() + 124 com.apple.securityd security_exception 1 Security 0x00000001943d66bc Security::MacOSError::MacOSError(int) + 340 com.apple.securityd security_exception 2 Security 0x00000001943d672c Security::MacOSError::throwMe(int) + 40 com.apple.securityd security_exception 3 Security 0x00000001942d28d4 Security::CodeSigning::KernelCode::locateGuest(__CFDictionary const*) + 1140 com.apple.securityd security_exception 4 Security 0x00000001942a7560 Security::CodeSigning::SecCode::autoLocateGuest(__CFDictionary const*, unsigned int) + 132 com.apple.securityd security_exception 5 Security 0x00000001942b05c8 SecCodeCopyGuestWithAttributes + 156 com.apple.securityd security_exception 6 support 0x0000000104d166e8 xpc_support_check_token + 280 com.apple.securityd security_exception 7 libxpc.dylib 0x00000001910632e0 _xpc_connection_check_peer_requirement + 428 com.apple.securityd security_exception 8 libxpc.dylib 0x000000019104b58c _xpc_connection_mach_event + 1056 com.apple.securityd security_exception 9 libdispatch.dylib 0x0000000105092c64 _dispatch_client_callout4 + 20 com.apple.securityd security_exception 10 libdispatch.dylib 0x00000001050b5994 _dispatch_mach_msg_invoke + 612 com.apple.securityd security_exception 11 libdispatch.dylib 0x000000010509bb98 _dispatch_lane_serial_drain + 368 com.apple.securityd security_exception 12 libdispatch.dylib 0x00000001050b70c4 _dispatch_mach_invoke + 496 com.apple.securityd security_exception 13 libdispatch.dylib 0x000000010509bb98 _dispatch_lane_serial_drain + 368 com.apple.securityd security_exception 14 libdispatch.dylib 0x000000010509ceb0 _dispatch_lane_invoke + 468 com.apple.securityd security_exception 15 libdispatch.dylib 0x00000001050ac958 _dispatch_root_queue_drain_deferred_wlh + 652 com.apple.securityd security_exception 16 libdispatch.dylib 0x00000001050abc30 _dispatch_workloop_worker_thread + 444 com.apple.securityd security_exception 17 libsystem_pthread.dylib 0x000000010513bd40 _pthread_wqthread + 288 com.apple.securityd security_exception 18 libsystem_pthread.dylib 0x0000000105143a94 start_wqthread + 8 com.apple.securityd esecurity_exception UNIX error exception: 1 com.apple.securityd security_exception 0 Security 0x00000001943d6108 Security::CommonError::LogBacktrace() + 124 com.apple.securityd security_exception 1 Security 0x00000001943d6464 Security::UnixError::UnixError(int, bool) + 352 com.apple.securityd security_exception 2 Security 0x00000001943d64d4 Security::UnixError::throwMe(int) + 44 com.apple.securityd security_exception 3 Security 0x00000001943d6aec Security::UnixPlusPlus::FileDesc::close() + 0 com.apple.securityd security_exception 4 Security 0x00000001942fcbd0 Security::CodeSigning::SingleDiskRep::fd() + 64 com.apple.securityd security_exception 5 Security 0x00000001942dfdfc Security::CodeSigning::MachORep::MachORep(char const*, Security::CodeSigning::DiskRep::Context const*) + 528 com.apple.securityd security_exception 6 Security 0x00000001942d223c Security::CodeSigning::KernelCode::identifyGuest(Security::CodeSigning::SecCode*, __CFData const**) + 684 com.apple.securityd security_exception 7 Security 0x00000001942a6bc8 Security::CodeSigning::SecCode::identify() + 100 com.apple.securityd security_exception 8 Security 0x00000001942a759c Security::CodeSigning::SecCode::autoLocateGuest(__CFDictionary const*, unsigned int) + 192 com.apple.securityd security_exception 9 Security 0x00000001942b05c8 SecCodeCopyGuestWithAttributes + 156 com.apple.securityd security_exception 10 support 0x0000000104d166e8 xpc_support_check_token + 280 com.apple.securityd security_exception 11 libxpc.dylib 0x00000001910632e0 _xpc_connection_check_peer_requirement + 428 com.apple.securityd security_exception 12 libxpc.dylib 0x000000019104b58c _xpc_connection_mach_event + 1056 com.apple.securityd security_exception 13 libdispatch.dylib 0x0000000105092c64 _dispatch_client_callout4 + 20 com.apple.securityd security_exception 14 libdispatch.dylib 0x00000001050b5994 _dispatch_mach_msg_invoke + 612 com.apple.securityd security_exception 15 libdispatch.dylib 0x000000010509bb98 _dispatch_lane_serial_drain + 368 com.apple.securityd security_exception 16 libdispatch.dylib 0x00000001050b70c4 _dispatch_mach_invoke + 496 com.apple.securityd security_exception 17 libdispatch.dylib 0x000000010509bb98 _dispatch_lane_serial_drain + 368 com.apple.securityd security_exception 18 libdispatch.dylib 0x000000010509ceb0 _dispatch_lane_invoke + 468 com.apple.securityd security_exception 19 libdispatch.dylib 0x00000001050ac958 _dispatch_root_queue_drain_deferred_wlh + 652 com.apple.securityd security_exception 20 libdispatch.dylib 0x00000001050abc30 _dispatch_workloop_worker_thread + 444 com.apple.securityd security_exception 21 libsystem_pthread.dylib 0x000000010513bd40 _pthread_wqthread + 288 com.apple.securityd security_exception 22 libsystem_pthread.dylib 0x0000000105143a94 start_wqthread + 8 com.apple.securityd <Missing Description> xpc_support_check_token: identifier com.example.app status: 100001 connection [0x13a904080] invalidated because the current process cancelled the connection by calling xpc_connection_cancel() com.apple.xpc <Missing Description> invalidate<com.example.app.service([app<application.com.example.app.271193970.274328866.AF06BFB0-EAE9-4539-AA9C-368A2584E9AE(501)>:46207])(501)>{vt: (null)}:

By “XPCService” do you mean an .xpc bundle within your app? Yes, I do. If so, applying a code signing requirement is kinda pointless. XPC services bundled within your app are only visible to your app. I'm not really sure what visibility means in this case, so asked here https://developer.apple.com/forums/thread/757627. Our security team requires us to make sure no-one else can connect to in case the can guess the service's port (if that's possible).

Thank you Quinn for looking into this. I think I found the problem: I'm having a simple scenario where main bundle contains and embedded XPCService bundle. In services NSXPCListenerDelegate I'm using the - setCodeSigningRequirement: on the incoming connection with valid code signing requirements. The connection was always refused and I've seen the security_exception in the log. So I compared the entitlements and signing of the applications bundles and only difference between working and non-working version was obvious presence of get-task-allow on developer signed builds and different signing certificate chain. So I was wrongly assuming that difference is due to signing. But it turned out that if I moved the problematic bundle to /Applications it immediately started to work. So it's not a signing difference but it's on-disk location difference. I think it can come from the application.sb: (allow file-read* process-exec ... (subpath "/Developer") (subpath "/Applications")) So when the XPCService is located under /Applications it can read the code signature from the parent bundle's executable. I'm not sure that profile is used for XPCServices bundled in the application but it would explain the behaviour.

@DTS Engineer I have a follow up question though. I understand that the bootstrap namespace is private to to application so I can't connect its XPC service by name if I don't have access to that namespace. But what If I know the mach port? Let's say I'll be lucky and guess the port :-) ...or use launchctl to tell me: $ launchctl print pid/12345/com.my.test.service ... endpoints = { "com.my.test.service" = { port = 0xeaf13 active = 1 managed = 1 reset = 0 hide = 0 watching = 0 } } ... In that case I would I be able to use msg_send() to send messages directly and eventually replicating what XPC does or is there any other level of protection (e.g. what comes to mind is that launchd can check the audit session or something)?

For you to have access to the port the kernel must have inserted an entry for that port into your task’s Mach port namespace. Which is exactly what the bootstrap service does when you successfully look up a name. Ah now I get it! I got stuck with deceiving idea that bootstrap service is kind of "DNS" for ports, but was not thinking for a minute how the port name actually gets into the process's port namespace. Now it makes much more sense. Thank you for connecting the dots for my dumb brain! :-)

Thank you for Kevin, I'll try to answer some questions What were the actual modifications that occurred and what files did they occur on? You said that specific bytes were modified but what was actually changed? The modification is always a replacement of meaningful ASCII string in Mach-O (main executable or dylibs). It's often a single sequence, but sometimes it's two where the second one is subsequence of the first. It's interesting that not all occurrences of the string are replaced in the whole file. I'm trying to make some sense out of it. It looks like the string is always replaced in both architectures in Mach-O, so it makes me think that whatever is doing that is aware of the executable format. Related to that point, how do you "know what you know" and are you sure you understand what actually happened? I get the zipped application bundles from customers and then compare the corrupted files to the bundles I extracted from pkg they use for install. Another clue I get is the install.log. In our postinstall we run gktool, so I see the output and it already complains there that bundle is altered. The component package which is getting corrupted has PackageInfo setup like this: Which forces installer to basically just replace the destination bundle instead of updating it so if the target bundle would be corrupted before installation it would not have any effect (unless it has a newer version) as installer would remove it and put in new bundle. That all leads me to fact that new bundle has to be corrupted either during or right after it was installed but before postinstall script runs.

Thank you Kevin, I'll file the incident in a few hours. I'll try to get some fresh sysdiagnose logs as well. We didn't collect those because most users are in managed environments so we need to arrange everything with their admins. Is it always the same "5f45 4545" value being replaced? No, it's same value per instance, but different for each instance. I have no idea why that would have happened, but it would be interesting to confirm whether or not that pattern holds more broadly. Definitely not all instances are replaced. I can still find same sequence in the file. I'll send you the originals and corrupted files so you can see for yourself. At the beginning it looked like all instances of the sequence were replaced but once we got more of these corrupted files it's clear that's not the case - we were just lucky with the first ones. Just to clarify, was that the specific installer pkg they had, not just the same version? I want to make sure you've ruled out the possibility that the bad file wasn't already inside the installer. I think I pretty much ruled this out. I was originally suspecting that admins were installing some modified packages so I asked if I can get those, but they were either links to our CDN or just unmodified packages.

So I found following being all over the logs when issue occurs: HALC_ProxyIOContext::IOWorkLoop: skipping cycle due to overload The machine is i9 with 64GB of RAM basically idling. Edit: Found another message which seems to lead to source of the problem: HALS_OverloadMessage: Overload due to client running as an adaptive unboosted So indeed it seems to be some priority problem?

Hello Kevin, I found the issue but I'll reply anyway as it may be useful to other people. Are you seeing this issue on later system versions? No, it looks like it's only Monterey, specifically the last two minor releases which I tested on. Is your XPC service directly presenting it's own window(s) or is this coming through your main app? Yes, I'm starting a NSApplication on main thread and the service listener on a second thread. That's only difference to regular XPCService template running just xpc_main(). If you're not already doing so, you may want to try having your app using "beginActivityWithOptions" to ensure that it's remaining active. I tried this both in application and xpc service and it didn't had any effect. I even tried task_policy, setting main thread to user interactive. Particularly on a old system like this, you may find that periodically "resetting" the message (say, every minute) works better than simply holding the same message "forever". I tried exactly this and it works. I was sending the reply block only at the start of the application to kickstart the service and boost it. Sending the block again before I want a playback seems to fix the priority problem on Monterey and doesn't have any side effects on newer systems. I still have a question though. Is there any recommendation how to investigate xpc service priorities (or QoS in general)? I tried launchctl print pid/13422/com.my.service and it gives handful of information but nothing I could figure out the boosting was not working correctly (spawn type seems to reflect only the startup priority). Also in Instruments I was using time profiler, system trace etc. and it shown the correct application lifecycle ("Foreground") but sound was still broken.