Wow, that’s quite a corner you’ve painted yourself in to (or maybe been painted in to :-).
Haha, yeah 😅 more so the former, luckily(?) - the joys of trying to pick up a several year old open source issue (definitely seeing why no one has done it yet)
Am I right in assuming that each mod’s entry point is in Java? So at some point the mod transitions from Java to native code? How does that transition happen? And could you hook that?
Yes. In general Java's System.load function is used to load it with JNI. (Well, more specifically, the mod probably is using some Java library it depends on that calls System.load rather than the developer directly using it, but from our perspective that should basically be the same thing.) And on hooking it...
If you could tweak that support then that’d provide an equivalent to your dlopen interpose, without the interpose.
Hmm, maybe. The two ways I can think of that we could hook at the Java level is either distributing modified JVM runtimes or maybe using Java agents/instrumentation (which I'm woefully unfamiliar with).
Though we do allow users to select their own JVM (sometimes different versions of the game need different JVM versions, and it can sometimes be useful to select specific versions to workaround certain bugs in certain cases), which would mean the second option is likely the better option.
This would, of course, fall down if a mod’s native code unpacks more native code, so I’m curious if that happens in practice?
I don't believe this really happens (I'm making an assumption based on the fact that most mods I've extracted only contain one library at maximum, so I don't think we'll see any "native loads more native" situations), so I think we can ignore this case. Though, in the rare case it does happen, we would intend to distribute both sandboxed and unsandboxed versions and allow users who really want to use these mods (or simply need to use mods that are fundamentally incompatible with the sandbox) to use the unsandboxed version.
Though, one question I have is that if I simply move the same implementation I have working (call to XPC service which checks that file is actually a library then removes quarantine, then proceed with library load) to move from a dlopen interposition to some Java instrumentation of System.load, wouldn't your comment about the overall system being brittle
That is, your overall system is gonna be quite brittle, and I suspect that interposition wouldn’t be the first thing to break O-:
still apply? Or are you just speaking in general, that if I had to do this hack, there's probably a lot of other, more fragile hacks in place too?