I was finally able to reproduce the issue relatively quickly on Ventura. Here is a report of my tests (sorry for the length of this post):
Installed Plugin with iLok protection : IRCAM-Trax from Flux Audio (demo version).
OS: macOS 13.2.1,
Hardware: iMac Retina 4K, 21.5-inch, 2019, 3,2 GHz Intel Core i7 6 cores
Sandbox options: Outgoing Connection • Audio Input • User Selected Files: Read-write • Download, Music and Movies folders: Read-write
Test 1: (DEBUG Scheme) SANDBOX, HARDENED RUNTIME (Disable Library Validation)
=> Failed: Exact same behavior as in macOS 12 (except that the error message is presented fourth instead of twice). The logs provided by the console wasn't very helpful, they don't seem related to the plugin crash.
Test 2: (DEBUG Scheme) SANDBOX, HARDENED RUNTIME (Disable Library Validation • Disable Memory Protection • Allow DYLD Environnement Variables • Allow Unsigned Executable Memory • Allo Execution of JIT-compiled Code)
=> Failed: Exactly as the previous.
Test 3: (DEBUG Scheme)
=> Failed: After deactivating the sandbox and Hardened Runtime, no plugin wants to open although all plugins have been validated again (since the Application Support folder where the referenced plugins are stored has been changed). I realized that the Hardened Runtime's ‘Disable Library Validation’ key is required to open any Audio Unit plugin, even in a non-sandboxed application (probably except those from Apple ? I didn’t test it).
Test 4: (DEBUG Scheme) HARDENED RUNTIME (Disable Library Validation)
=> Failed: Here the iLok manager finally complained about the fact that it detected a debugger when I tried to open Trax… impressive.
Test 5: (RELEASE Scheme) HARDENED RUNTIME (Disable Library Validation)
=> Crashed: Strangely, with the release scheme, the application crashed when I tried to open Trax, I join the report below:
Anonymous-08:09:2023.crash
Test 6: (RELEASE Scheme) HARDENED RUNTIME (Disable Library Validation • Disable Memory Protection )
=> Success: Trax did open and all work as expected!
I was happy to finally find a solution, but the bad news is that this solution involves not using a Sandbox, which is not an option for me since I would like to distribute on the AppStore.
I still wondered how Logic and Garage Band do to work around this issue, so I tested again with Garage Band (on Monterey, the test was done with a licensed version of Logic Pro 10.7, which I think is sandboxed, and it had worked normally). But, finally, Garage Band 10.4.8 behave exactly the same way than my application:
First this message this message was displayed 4 times when I tried to open Trax:
Then, the error 100001 (reproduced in a previous post) was displayed.
Next, this more informative message was displayed:
Finally, you can see the content of the Garage Band’s temporary folder on this last screenshot: each error produces a file with another name and don’t overwrite the previous one. I hope this folder is cleared regularly because, depending on the size of the bundle, and how many times the user gets this error, this could consume some hard drive space unnecessarily...
So, it finally appears that there is a real conflict, at least from macOS 12, but it could earlier, between some plugins authorized via iLok and the OS (with the collection of my beta-tester, it was something like 75% of the iLok protected-plugins). The only remaining doubt is about the try we made with success on Logic Pro 10.7.9 and Monterey (but I have no prove that this version is sandboxed, the simplest explanation would be that it’s not...). Do you think there is something else I can do to find a solution ? It seems that these problems are far removed from my application and that I don't have much control over them. Thanks!