Apologies I meant to ask about activity tracing as well (they are related to this topic but if we think another forum post is better I can do it too). At certain point this also became one of the directions we wanted to explore but we've only uncovered more questions. The related documentation stated something but very vaguely. https://developer.apple.com/documentation/os/generating-log-messages-from-your-code?language=objc#Choose-the-Appropriate-Log-Level-for-Each-Message If an activity object exists, the system captures information for the related process chain We had hoped that this would somewhat play into the "speculative logging" approach we had touched upon, in the sense that if we try to log an error or fault within an activity, then it helps to capture and persist other logs on the activity chain even though they are originally not meant to be. But unfortunately from our test it didn't seem to be behaving towards that understanding. Then our question is, if we may ask - what are the exact additional information the system captures "for the related process chain"? Are activities always persisted and would they also contribute to log quarantine? Our thoughts were that if we can, we'd like to create an independent activity for each event we receive from the system so the logs on every event are automatically correlated (please refer to the same use case described in FB21839588). However as demonstrated in the FB ticket, that would mean A LOT of events and hence a lot of activities. I noticed we do have a flag Signpost-Persisted to control persisting for signposts, but there isn't a control for activities. My assumption is that they (the activities themselves) are always to be persisted so they would indeed contribute to quarantine in the worst case, is that correct? (Although from log stats it looks like each individual activity is tiny in terms of storage size, so maybe they are not a big concern themselves?) We noticed this control flag Propagate-with-Activity used by some logging configuration files from Apple. We didn't find any official Apple documentation but some 3rd party MDM vendors mentioned something about Propagate-with-Activity Messages attached to the activity tree Messages are attached to the activity tree in Console and crash dumps Right now based on our observation messages are always attached to the activity tree in Console regardless, and we can't seem to be able to find anything to do between activities and crash dumps. Maybe this flag is obsolete?

While this isn't the answer I had been wishing for it does give me a lot of food for thought. Thanks The Eskimo for looking into it! [quote='874248022, DTS Engineer, /thread/814022?answerId=874248022#874248022'] When it rotates a file, the system calculates how fast it filled up. [/quote] So wrt this note, I understand it's more complicated than just rotating a single file, but still it sounds like everyone writing into the Unified Logging system can contribute to this. It strikes me at least one strategy (if not a preferable one), is to disable logging for all other processes/subsystems (even some of the native OS processes) that tend to be noisy too? Of course this is meant as a temporary and desperate measure, in theory it should raise our survival chance? And another thought, what if we disable persisting, but then have an external party help redirect our logs to be persisted? I first thought the xctrace record --template "Logging" might be a good candidate for this task but later found out the "deferred", or "windowed" log tracing seemed to be taking only from the snapshot too. If we'd run a deferred log tracing for an hour, it would likely only save the last 5 minutes out of the snapshot. Only the "live" (i.e. the "immediate") log tracing can help persist log events it has intercepted. But conceivably live log tracing would have an even worse performance penalty since it attempts to remodel the events in real time? I've seen errors reported by the Instruments app when doing a live log tracing that due to the sheer number of events it had to drop some of them from time to time. And what of log stream ... > persisted_logs.log? Even if performance impact introduced by live log intercepting and file writing can be deemed acceptable (as long as this is only, again, a desperate measure), would this one also suffer from dropping events? And lastly, I filed an improvement request anyway (FB21839588). I think this problem is practical enough, in the sense that if we do a persist:debug for our heaviest component, we deterministically get quarantined. That part in itself is not a hypothetical question (but I do agree the hypothetical part is that we don't exactly know if there's a concrete case that would actually require us to enable persist:debug in the first place). I attached a logarchive example and its log stats output in the FB ticket to demonstrate our current use case and why we hit log quarantine seemingly easily.

Hi @eskimo a follow-up question here.. I fully understand that Apple doesn't want to divulge whether an enhancement request has been prioritized, or when it can be available. For now if our product decides that the capability of registering a global launch agent is important to us, that we'd like to keep doing the "launchctl" way, is that still accepted by Apple? I did notice a line on this page that says "In apps that target macOS 13 and later, your app needs to only use the property list locations outlined above" and my own interpretation was that if our app's minimum deployment target was marked as 13.0, then we absolutely had to switch over to the SMAppService interface. Did I misinterpret that? On the other hand, can we lower our app's minimum deployment target to macOS 12.0 to circumvent that restriction?

Sounds good thanks The Eskimo! FYI one of the use cases I can definitely think of is - some organizations may have SOE images with a user 501 like "soe_admin", who is responsible of pre-installing all software required. For our software, the installer can trivially register a login item or launch agent bound to the current user (501) using SMAppService via a postinstall script. However the actual device users will not be 501 so the login item or launch agent will not load properly for them unless they manually intervene (which we don't want because our software is largely silent/background). Our software also has a launch daemon, which is loaded cross-user properly. I thought of using the daemon helper to ensure the login item/launch agent is registered properly across different gui sessions - but alas the daemon runs under the system domain, so it cannot read or write the user/gui domain (from SMAppService's header file I understand this is by design). So unless we have the proper support of global login item/launch agent I think we'll be forced to detour. Feedback# is FB13137999, I've included the use case above in it as well.

Thanks The Eskimo! I've submitted FB11985759.

It's using the NEPacketTunnelProvider API, which creates a VPN profile in System Settings > Network. The first time when our app creates that profile, we see a prompt saying "App XYZ would like to filter network content. Allow/Don't Allow", but that's not what we are talking about here. Instead let's assume we've already allowed the profile to be created, and then what happens is that (1) when our code is operating on that profile, e.g. starting/stopping the VPN connection, or (2) the user is manually operating on that profile, we have a chance to see this new popup in question. And also it only happens if we currently have the System Settings > Network > VPN & Filters tab opened (again this is Ventura only).

Thanks Jason, I posted a question there too. However, I wanted to say the reason I put it here (dev forum) was because our software has a VPN functionality and we are seeing this popup on Ventura and know it's gonna impact our customers' user experience. For a few reasons I also posted in the question on the Support website, I really suspect this is a Ventura bug. The popup isn't from a 3rd party VPN software, it's from an Apple process called "VPN" (/System/Library/ExtensionKit/Extensions/VPN.appex). The popup DOESN'T happen if I do not open System Settings > Network > VPN & Filters settings tab in the first place. For example, if I close the System Settings dialog, then no matter how I manipulate a VPN software, I won't see this prompt. The popup DOESN'T change anything either - no matter if I accept it or just cancel it, 3rd party software can still manage its VPN profile just fine. When we allowed the VPN software to create a VPN profile in the system in the first place, we already saw another system prompt asking for user consent to allow this app to monitor network traffic. Right now I'm very sure our software is not trying to do any modifications (not trying to create any new VPN profile), it's only trying to start or stop the VPN connection. It doesn't seem to be a very good user experience if I got a prompt out of blue every time when trying to connect or disconnect the VPN (even though the popup doesn't really block our functionality).

Watched and learned. Thanks The Eskimo, always a pleasure!

Happy New Year eskimo! The crash logs we collect from our customers are generated by PLCrashReporter I believe. It somehow yanks the system symbols out (sounds like a PLCrashReporter thing) and we don't have a convenient way to re-symbolicate that part. That said I'm fairly confident if we would symbolicate output_18765164.log it would produce exactly the same crashed stack as the bash one (apart from the fact of different parent processes). One more detail we recently figured out - we noticed that currently our crash reporter has a restriction built in our code that it's only able to properly process & upload crash logs if it's running with root access. Since we are getting these crash logs, it proves those processes have been launching as the root user, which seems to conflict with the fact that they are (at least supposed to be) registered as Launch Agent and should be running as the current logged in user? Again to me it still doesn't pinpoint the issue any further than our current speculation - it could still be a system bug/software bug/user tampering, but I thought to mention this new detail.

Radar ticket for this - FB9800999

SessionGetInfo sounds like a great idea! I'll forward it to our team. can you post a full crash report? Yes I can help with that. We already have a radar ticket on the original AppKit crash. Let me ask my colleague who filed it about the number and get back here. It's around the holidays though I might not be able to reach him soon. And in the likely case I don't post any feedback until New Year just want to wish you Happy Holidays Eskimo (and of course to anyone who stumbles on this post around this time of year), your help and advice are always appreciated!

Thanks for the pointer to the Technote Eskimo! It was good reading. I presume that you’re not in touch with any of these users? That is unfortunately true. From crash logs alone we currently have no means to backtrack to the specific users. We'll have to wait for the users to come contacting us instead. Given that the crash is likely only happening without a GUI session it's likely they would not notice it for quite a while (or maybe not at all). For now I think my interpretation of your answer is that from the OS point of view, there is no official way (at least no known one) to be able to launch an agent under the Aqua type (and yes our launchd.plist doesn't specify LimitLoadToSessionType) while denying it access to the graphic context. So the users must have somehow either "monkeyed" with the system, or like you suggested, with our config. Assuming our hypothesis is true and there is no good option to prevent end users from doing what they are doing, currently our team has come up with an idea of detecting this situation (no GUI context) during our agent's startup in hoping to skip the entire AppKit entry call. Specifically we are thinking of calling CGWindowListCopyWindowInfo(kCGWindowListOptionAll, kCGNullWindowID) before we engage all UI related stuff. We verified that if we call this without graphic context we would get nil while otherwise we would get an array with something, which serves as an indicator to us. Does this sound like a practical idea or is there perhaps anything we overlooked?

Thank you The Eskimo as always!

Answering my own question here in case anyone comes across the same. I figured out this crash happens if we call [[NSStatusBar systemStatusBar] statusItemWithLength:NSSquareStatusItemLength] when a graphic session is not available. E.g. if the code is executed through an ssh session without the same user on any active GUI session. Interesting that I tried the same with older macOS versions I don't see the crash (while I do see some error messages printed on console). It matches the fact that our crash reports have all been coming from macOS 12. What I still don't understand is that how come our code has been executed if there is no GUI (it is registered as a LaunchAgent) but I've opened a new post for that discussion (https://developer.apple.com/forums/thread/696859).

Thanks Matt! Looks like it is not enforced for remote connections either. Filed a ticket FB9716553. From some logs we came through it looks like the Network framework tried to upgrade the connection to HTTPS and failed (expected because we didn't enable HTTPS on the target host during the test) and then it just fell back. Still to our own concern - being this close to release we don't expect this behavior to change in GA right? ;p