Thank you Kevin for these helpful details. I'm still catching up with the linked documents and I'll respond once I have gone through those.

That "send0" is implemented by the JDK by invoking the sendto() system call. In this case, the sendto() is returning a EHOSTUNREACH error which is what is then propagated to the application. I would like to note that this (and one other error message that I'm investigating), I feel, are a bit misleading. When these issues showed up several weeks back, I and others started looking into this. At first we spent several days trying to understand if this was something to do with the networking configurations on the host or other devices. We had to involve some network admins from our lab to try and debug this. After several days of investigation, we came to realize that the "Local Networking" restrictions put in place in 15.x of macosx are at play.

I asked about that internally and we suspect that it’s caused by one of these internal identities being cloned with an override of the audit user ID being set to 0. However, I’m able to take this further at this time. The runningboardd implementation is complex and to learn more I’d need to consult with the folks who maintain that, and I don’t want to do that until I know more about your side of this equation. Thank you Quinn for looking deeper into this. I have been trying to reproduce this locally but so far I haven't been able to. This is still on my TODO list to investigate and I hope to get back shortly with additional details when I have them. Just to clear, the presence of this log message does indicate that there is some genuine issue that is worth investigating, right?

Here's what I understand of the log messages that I pasted in my previous reply. Remember that the process id of interest is 58700. In the log above, you will notice that the only place where this proces id 58700 is present is the line: cfprefsd [0x74c933e80] activating connection: mach=false listener=false peer=true name=com.apple.cfprefsd.daemon.peer[58700].0x74c933e80 That line isn't too interesting and I don't know if it's relevant in this discussion, so I'll skip that one. The good thing however is that these logs appear to have captured enough details about the "Local Network" restriction's implementation. Several of those above log messages have something interesting, and I think it can be summarized by these few: UserEventAgent Got local network blocked notification: pid: 7384, uuid: 4E7709E7-AD5C-38B8-9ED0-0354767877BD, bundle_id: (null) UserEventAgent LocalNetwork: found bundle id marco-foobar by PID UserEventAgent LocalNetwork: did not find bundle ID for UUID 4E7709E7-AD5C-38B8-9ED0-0354767877BD UserEventAgent Found bundle ID: marco-foobar nehelper application record search init. Node: (null) bundleID: <private> itemID: 0 ... 708 info 2025-03-13 09:53:42.446497 +0000 runningboardd _executablePath = /opt/marco-foobar/start-marco-foobar.bash 708 info 2025-03-13 09:53:42.446500 +0000 runningboardd no additional launch properties found for <private> 708 default 2025-03-13 09:53:42.446543 +0000 runningboardd _resolveProcessWithIdentifier pid 7384 euid 0 auid 0 708 default 2025-03-13 09:53:42.446584 +0000 runningboardd Resolved pid 7384 to [osservice<polo-foobar>:7384] ... runningboardd [osservice<polo-foobar>:7384] is not RunningBoard jetsam managed. runningboardd [osservice<polo-foobar>:7384] This process will not be managed. runningboardd PERF: Received request from [osservice<com.apple.nehelper>:726] (euid 0, auid 0) (persona (null)): lookupProcessName:error: nehelper No team ID found for (bundleID: marco-foobar, name: marco-foobar) So when the java program (in process 58700) intitated the sendto() local network operation, it appears to have triggered a "local network blocked notification". I'm not sure if that log line from UserEventAgent means that the notification pop-up (asking the user to allow/disallow the operation) was generated or whether it is determining if the pop-up needs to be generated. In any case, that log message indicates that the pid is 7384. Looking at the additional data I collected from that system and the documentation of "Local Networking" which states: When a process performs a local network operation, macOS tries to track down the responsible code. For example, if your app spawns a helper tool and the helper tool performs a local network operation, macOS considers the app to be the responsible code. So what that log tells me is that the local network restriction has identified 7384 process as the top level root application for the 58700 java process which is doing the networking operation. So I went back and looked into the process and launch hierarchy on this setup. 7384 is a process launched through launchd with the /opt/marco-foobar/start-marco-foobar.bash bash script file as the executable. When I say launchd, I may not be using the right term here. I think the /opt/marco-foobar/start-marco-foobar.bash bash script gets launched whenever that macos system starts (I will confirm that today). The plist file which configures this bash script as the entry point looks something like: <plist version="1.0"> <dict> <key>Label</key> <string>polo-foobar</string> <key>ProgramArguments</key> <array> <string>/opt/marco-foobar/start-marco-foobar.bash</string> ... and launchtl list on that system shows: PID Status Label ... 7384 0 polo-foobar ... The /opt/marco-foobar/start-marco-foobar.bash ultimately ends up doing a: ... exec /opt/polo/bin/polo-foobar polo-foobar is a 3rd party binary and doesn't have any plist file of its own: launchctl plist /opt/polo/bin/polo-foobar 64-bit Mach-O does not have a __TEXT,__info_plist or is invalid. polo-foobar is a generic (3rd party) framework which allows application specific code to be executed. In this case, it ends up launching a java process which then further launches (through Java's ProcessBuilder API https://docs.oracle.com/en/java/javase/23/docs/api/java.base/java/lang/ProcessBuilder.html#start()) the ultimate 58700 java process. So given all this, it seems to me that the entry point process /opt/marco-foobar/start-marco-foobar.bash (which "exec"ed the polo-foobar binary) is being denied access to the network operation. Did I understand it right? If so, how do I go about addressing this issue. The local networking document states: If you ship a launchd agent that’s not installed using SMAppService, make macOS aware of the responsible code by setting the AssociatedBundleIdentifiers property in your launchd property list. Does that mean I need to add the AssociatedBundleIdentifiers property to the plist snippet that I shared previously? What value (values?) would I add to it? Furthermore, these processes are running on a system which acts like a "server" and there's no user interaction involved. So what are the options of making this "allow this process (sequence) access to networking operations" non-interactive and configurable/automatable? While at it, I would like to note that the output of ps -Meo pid,pcpu,cputime,start,pmem,vsz,rss,state,wchan,user,args for this top level 7384 process which seems to have been denied the network operation access shows that it is running as root: USER PID TT %CPU STAT PRI STIME UTIME COMMAND PID %CPU TIME STARTED %MEM VSZ RSS STAT WCHAN USER ARGS root 7384 ?? 0.0 S 20T 0:00.03 0:00.55 /opt/ma 7384 0.0 73:40.19 1Mar25 0.4 412145952 58976 S - root /opt/polo/bin/polo-foobar So this local network access denial for this process seems to go against what the local networking documentation states: macOS considerations macOS maintains separate local network privacy state for each user account. macOS automatically allows local network access by: Any daemon started by launchd Any program running as root ... Overall this feels way too complicated to manage and if I understand it correctly, none of these issues has to do with java itself and I can imagine the exact same launch sequence leading to a go (or even python) application which uses that language's standard networking APIs to run into this same thing. Have I misunderstood this? While at it, I would also like to understand if those above log messages show any other issues that might need to be addressed.

Here's the relevant system logs from one such failing setup: PID Type Date & Time Process Message 706 default 2025-03-13 09:53:42.297782 +0000 cfprefsd [0x74c933e80] activating connection: mach=false listener=false peer=true name=com.apple.cfprefsd.daemon.peer[58700].0x74c933e80 default 2025-03-13 09:53:42.337316 +0000 kernel AppleAVD: addFrameWorkLoad(): clientID 0 clientFPS too slow! Forcing VMAX! rollingAvgFPS: 186 < streamFPS: 240 c[0].activeLoadRate 1990656000 c[0].clientCount 1 minSpeed 1 default 2025-03-13 09:53:42.391103 +0000 kernel AppleAVD: addFrameWorkLoad(): clientID 0 clientFPS too slow! Forcing VMAX! rollingAvgFPS: 186 < streamFPS: 240 c[0].activeLoadRate 1990656000 c[0].clientCount 1 minSpeed 1 638 info 2025-03-13 09:53:42.444295 +0000 UserEventAgent Got local network blocked notification: pid: 7384, uuid: 4E7709E7-AD5C-38B8-9ED0-0354767877BD, bundle_id: (null) 638 default 2025-03-13 09:53:42.444323 +0000 UserEventAgent LocalNetwork: found bundle id marco-foobar by PID 638 info 2025-03-13 09:53:42.444489 +0000 UserEventAgent LocalNetwork: did not find bundle ID for UUID 4E7709E7-AD5C-38B8-9ED0-0354767877BD 638 info 2025-03-13 09:53:42.444489 +0000 UserEventAgent Found bundle ID: marco-foobar 726 info 2025-03-13 09:53:42.444717 +0000 nehelper application record search init. Node: (null) bundleID: <private> itemID: 0 708 info 2025-03-13 09:53:42.445100 +0000 runningboardd PERF: Received request from [osservice<com.apple.nehelper>:726] (euid 0, auid 0) (persona (null)): lookupHandleForPredicate:error: 708 default 2025-03-13 09:53:42.445107 +0000 runningboardd PERF: Received lookupHandleForPredicate request from [osservice<com.apple.nehelper>:726] (euid 0, auid 0) (persona (null)) 708 info 2025-03-13 09:53:42.445481 +0000 runningboardd [osservice<com.apple.nehelper>:726] handle lookup could not find a matching process 708 info 2025-03-13 09:53:42.445554 +0000 runningboardd Error handling message from [osservice<com.apple.nehelper>:726]: <Error Domain=RBSRequestErrorDomain Code=3 "Specified predicate did not match any processes" UserInfo={NSLocalizedFailureReason=Specified predicate did not match any processes}> debug 2025-03-13 09:53:42.445918 +0000 kernel igmp_append_relq: adding inm e76b811a46a86ff3 on relq ifp en0 debug 2025-03-13 09:53:42.445930 +0000 kernel igmp_append_relq: adding inm e76b811a46a86f13 on relq ifp en0 debug 2025-03-13 09:53:42.445954 +0000 kernel igmp_flush_relq: flushing e76b811a46a86ff3 on relq ifp en0 debug 2025-03-13 09:53:42.445957 +0000 kernel igmp_flush_relq: flushing e76b811a46a86f13 on relq ifp en0 708 info 2025-03-13 09:53:42.446219 +0000 runningboardd PERF: Received request from [osservice<com.apple.nehelper>:726] (euid 0, auid 0) (persona (null)): lookupHandleForPredicate:error: 708 default 2025-03-13 09:53:42.446224 +0000 runningboardd PERF: Received lookupHandleForPredicate request from [osservice<com.apple.nehelper>:726] (euid 0, auid 0) (persona (null)) 708 info 2025-03-13 09:53:42.446495 +0000 runningboardd _multiInstance = 0 708 info 2025-03-13 09:53:42.446497 +0000 runningboardd _executablePath = /opt/marco-foobar/start-marco-foobar.bash 708 info 2025-03-13 09:53:42.446500 +0000 runningboardd no additional launch properties found for <private> 708 default 2025-03-13 09:53:42.446543 +0000 runningboardd _resolveProcessWithIdentifier pid 7384 euid 0 auid 0 708 default 2025-03-13 09:53:42.446584 +0000 runningboardd Resolved pid 7384 to [osservice<polo-foobar>:7384] 708 error 2025-03-13 09:53:42.446629 +0000 runningboardd memorystatus_control error: MEMORYSTATUS_CMD_CONVERT_MEMLIMIT_MB(-1) returned -1 22 (Invalid argument) 708 error 2025-03-13 09:53:42.446631 +0000 runningboardd memorystatus_control error: MEMORYSTATUS_CMD_CONVERT_MEMLIMIT_MB(0) returned -1 22 (Invalid argument) 708 default 2025-03-13 09:53:42.447432 +0000 runningboardd Full encoding handle <private>, with data 47a8317e00001cd8, and pid 7384 708 default 2025-03-13 09:53:42.447678 +0000 runningboardd [osservice<polo-foobar>:7384] is not RunningBoard jetsam managed. 708 default 2025-03-13 09:53:42.447686 +0000 runningboardd [osservice<polo-foobar>:7384] This process will not be managed. 708 info 2025-03-13 09:53:42.448200 +0000 runningboardd PERF: Received request from [osservice<com.apple.nehelper>:726] (euid 0, auid 0) (persona (null)): lookupProcessName:error: 726 info 2025-03-13 09:53:42.448319 +0000 nehelper No team ID found for (bundleID: marco-foobar, name: marco-foobar) This log was captured when the java process 58700 initiated the sendto() and ran into an exception. That's the entirety of the log during that duration and I haven't left out anything.

Hello Quinn, IMPORTANT The following contains a lot of implementation details. Don’t encode this knowledge into a product that you ship to a wide range of users. It has changed in the past and will likely change again in the future. I understand. I suspect the identity it is talking about is the one explained as designated requirement [in TN3127] That’s not the case. Rather, this is an internal identity used by runningboardd to classify the processes it knows about based on how they got started. Thank you for clarifying that. That’s because starting the .idleDisplaySleepDisabled activity has to raise an assertion [1] with runningboardd, which causes the process to check in with it, Would you happen to know what activities (or system calls or library functions?) would make a java process raise an assertion with runningboardd? As to how your process got two identities, I’m not 100% sure. The AUID refers to the audit user ID, that is, the value returned by getauid. I wasn’t able to determine what the (0) means. I suspect that this is due to the weird way that Java apps launch themselves, often involving things like a fork without an exec, or vice versa. However, I played around with various techniques like that and wasn’t able to reproduce it. Indeed, there are several ways to launch a Java program but most of times the entry happens through the java executable. Java standard APIs also has a java.lang.ProcessBuilder class which allows for a running Java program to launch any other process (and that launched process can be itself be a Java program) https://docs.oracle.com/en/java/javase/23/docs/api/java.base/java/lang/ProcessBuilder.html#start(). On macos platform, ProcessBuilder.start() ends up calling posix_spawn system call to launch the new process. I will take a deeper look today/tomorrow to see if I can reproduce these logs on my local setup with some experimental java applications.

Yes, I would interested in those [JIT] details. OK. I’d like to split that off to a separate thread, because it has a bunch of backstory and general complexity. I also want to call in some help from a colleague who tracks this issue more closely than I do. Please start a new thread in App & System Services > Core OS topic area. Post the thread URL in your next reply here; that way I’ll be sure to see it. Done - here's the new thread https://developer.apple.com/forums/thread/776290

com.apple.developer.networking.multicast One of my colleagues has already pointed you at TN3179 Understanding local network privacy, which has a bunch of info on this topic. As to whether this entitlement is required on macOS, that’s definitely not the case. TN3179 says that, but you have to join the dots. The “macOS 10.0+” at the top of this page is just wrong. I filed a bug and we fixed that a while back (r. 105890246). Sadly, that patch seems to have come unstuck, so I’ve filed a new bug (r. 146015779). Thank you very much. For me that was a major source of distraction and confusion while investigating some multicast failures, because I wasn't sure if I should focus on this entitlement or look elsewhere during the investigation. This confirmation helps.

The hardened runtime enables a bunch of additional security checks. None of them are related to networking. This is good to know. Thank you. Some of them are very important to a Java VM author, most notably the com.apple.security.cs.allow-jit -> com.apple.security.cs.allow-unsigned-executable-memory -> com.apple.security.cs.disable-executable-page-protection cascade. My advice on that front: This sequence is a trade off between increasing programmer convenience and decreasing security. com.apple.security.cs.allow-jit is the most secure, but requires extra work in your code. Only set one of these entitlements, because each is a superset of its predecessor. com.apple.security.cs.disable-executable-page-protection is rarely useful. Indeed, on Apple silicon [1] it’s the same as com.apple.security.cs.allow-unsigned-executable-memory. I went back and looked at the entitlements we apply for a notarized package of the JDK. In addition to some other entitlements, it currently includes both com.apple.security.cs.allow-jit and com.apple.security.cs.allow-unsigned-executable-memory. If you want to investigate moving from com.apple.security.cs.allow-unsigned-executable-memory to com.apple.security.cs.allow-jit, lemme know because there are a bunch of additional resources on that topic. Yes, I would interested in those details. Actually, the JVM team of the JDK would very likely be interested in that detail (JVM isn't my area of expertise). But yes, any additional details and recommendations that you can provide on moving away from any of these entitlements is very much useful for us. [1] I’m talking about native code here. If you’re running under Rosetta and thus generating Intel code, this distinction is still relevant. I'm pretty sure we don't support Rosetta for Apple Silicon (I will check with others). The JDK does support x86_64 of macos and from what I see we don't use the com.apple.security.cs.disable-executable-page-protection entitlement in any of these platforms. Reading through the documentation of com.apple.security.cs.disable-executable-page-protection https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.security.cs.disable-executable-page-protection, it states: The system causes an app that attempts to directly modify sections of its own executable files on disk to forcefully exit. Use the Disable Executable Memory Protection Entitlement to enable this kind of unsafe software update. The JDK doesn't do such modifications. So I think that's understandable why we don't use that entitlement.

if the executable hasn't been configured with a relevant entitlement, what happens when the executable invokes on such operation. That varies by API: Most APIs just fail, returning some sort of error. That’s certainly the case for the entitlements you mentioned. Some APIs trap. Some APIs silently do nothing. Or log something and then do nothing. Thank you for that detail. I was hoping there would be a central place to find some logs/hints to correlate some odd failure resulting out of a missing entitlement. Is it even technically feasible for macos to generate such logs and if yes, do you recommend filing a feedback assistance issue for it?

For the JDK, I'd be somewhat surprised to see a lot of multicast networking (as an example) being performed by javac itself, but of course applications developed with the JDK that use the Java APIs to do networking operations from java could. Right - javac executable wouldn't be involved in many of these networking calls. Like you note, java would. Thank you both of you for these extensive details about local network changes. I'm reading through the technote and other linked material. I'll experiment a bit and come back with additional questions. These replies so far has helped me understand the relevance of entitlements, so thank you very much.

Hello Quinn, There isn’t a good way to answer this. You’re talking about a span of two or three major OS releases, each with a bazillion changes that could introduce a performance regression in this space. I understand. I wasn't too sure if I wanted to raise this question in first place, given that I don't have anything concrete to follow up on. I was merely hoping that the stack frames might provide some hints on whether or not whatever is happening in that call stack should be happening at all. But yes, right now I don't have anything concrete. I'll keep an eye to see if any of this warrants a deeper investigation. I realise that your specific setup is gonna end up using dlopen a lot. I wonder if you could do something with the new mergeable libraries feature to package your runtime up with all the native modules into a single app executable. I suspect that’d seriously help with launch times. An Apple Library Primer has links to info on mergeable libraries. Thank you for that link, I'm going to read up on it and discuss with my team mates more familiar in this area to understand if there's anything we could do with that feature.

No correction necessary. That's all correct. Something complicated like the JDK would likely need entitlements for the hardened runtime if notarization was desired. Thank you for those inputs. But you're starting to get into obscure technical details here. Why do you care about the JDK anyway? I contribute to the OpenJDK project and in recent releases of macos, we have been seeing various odd networking failures on macos, involving Java applications that do very basic operations with sockets. They are hard to debug or narrow down. So I'm going through the macos manuals and trying to understand some of the changes in the recent releases and the general guidelines about packaging applications. Some of these failures I believe are related to the recent "Local Network" restrictions which have been introduced in 15.x of macosx. I will create separate thread for that one, but before getting to that, I wanted to make sure I understand some of these existing primitives in macos. Thank you for your helpful inputs so far.

Entitlements only apply to the sandbox. They "entitle" the sandboxed app to break free of the sandbox for very specific needs. Thank you @Etresoft for that input - that was useful. I now read up on App Sandbox in macos here https://developer.apple.com/documentation/Security/app-sandbox. From what I understand of that, sandboxing an application is a choice that's left to the application itself. So if an application isn't sandboxed then access to resources/operations (like the networking operations) aren't enforced at runtime through entitlements. An application is mandated to be sandboxed if the application is distributed through the Mac App Store. Given all this, it then means that the entitlements enforced for a sandboxed app aren't applicable to the JDK (since it isn't sandboxed). Having said that, while reading through that doc, I found another construct in macos called the "Hardened Runtime" https://developer.apple.com/documentation/security/hardened-runtime. This too states that applications decide whether they want hardened runtime. However, importantly, and something that's relevant for the JDK, that document states that for an application that will be notarized (the JDK will be), it must enable the hardened runtime. That same page mentions that when an application is run with hardened runtime then certain entitlements are necessary if the application does some specific operations. That page lists the exact entitlements necessary for a hardened runtime and in context of the JDK, some of them are relevant. In that list, I don't find any networking related entitlements. So I take it that an hardened runtime doesn't enforce restrictions on networking resources/operations through entitlements and thus the JDK doesn't have to apply for these networking entitlements when running in a hardened runtime. Again, please correct me if I got this wrong. Thanks again for the help so far.

If we build it with -mllvm -enable-constraint-elimination=0 it will remove the issue. @y-c-c, thank you for that input and the discussion on the github thread where you provided the reproducer https://github.com/Homebrew/homebrew-core/issues/195325. I have now verified that using -mllvm -enable-constraint-elimination=0 does indeed prevent this issue and the OpenJDK build succeeds (a very minimal test of using the built binary seems to work fine too). I have tried those flags with both -O2 and -O3 and it prevents this issue for both those optimization levels. I've added this detail to the Apple feedback issue FB15162411. Hopefully this will help address the issue in some upcoming release of XCode. Thank you everyone for all the help in trying to narrow this down.