I am working on an application with multiple helper apps and tools with a fairly nested structure Why? I'm not even sure this will work at all via the Mac App Store. The documentation says you can have other, standalone executables in the MacOS folder, but I don't know if other app bundles qualify.
Technical Note TN2206 also specifically says you can have "helper apps and tools" inside the "Contents/Helpers" directory. Again, I'm not sure about actual app bundles though. Even if it is allowed, it seems to be risky. Few people do this. So if there is a problem, you are the only one who will notice.
Could these apps be changed to single-file tools or XPC bundles?
It is good that you are comparing to what Xcode does. Does that mean you aren't actually using Xcode? That's a Red Flag. But if you can hack up a reasonable facsimile of your app and observe how Xcode signs it, you should always copy what Xcode does. Ignore what you see on the internet about it unless it was written by eskimo. After checking, I see that you are actually referencing one of eskimo's how-to threads. So again, you're on the right track. In that thread, eskimo is specifically talking about signings a complex executable not using Xcode. You haven't posted all of Xcode's operations, but it looks like Xcode is doing the same thing by signing a deeply tested framework tool first.
That's a more difficult question. I don't really know the answer. Normally, for the Mac App Store, you just sign your app and submit. Then Apple re-signs and publishes. If you aren't using Xcode, then you are opening yourself up to the possibility of a code-signing error on submission. Hopefully the stand-alone submission tool will catch that problem. If not, hopefully App Review will catch it.
Mac App Store apps don't have to be notarized, or even use the hardened runtime. But if you want to turn all those things on anyway, it won't hurt. Plus, it seems logical that the hardened runtime will be required in the Mac App Store at some point. Perhaps it already is and I just haven't noticed it. My Mac App Store app has the hardened runtime enabled. I think I must have done that myself when I started notarizing the non-Mac App Store version, just to be on the safe side.
I don't know the exact answer for this either. However, it should be easy enough to test. I can tell you that you shouldn't use symlinks. Even if it worked, don't do that. The example that you would want to follow here is LoginItems. LoginItems are full app bundles stored in "Contents/Library/LoginItems" of your app. While I don't use them in my own app, I have other apps that do. And I can see that those apps definitely share frameworks with the top-level app. The tricky part is the linking. When I do an "tool -L" on the login item, it reports that the frameworks are referenced directly from "@rpath" without any ".." parent links. So this LoginItem executable has the correct RPATH encoded in it. It is possible that using ".." might still work, but I strongly recommend using the correct RPATH. This might require some research. It is the kind of thing that Xcode does for you. If you are doing it on your, you will have to figure it out on your own. In terms of tricky linking, RPATH is where it's at.
You may need to use the "install\_name\_tool" tool to set those paths inside the executable. You might need the "-headerpad\_max\_install\_names" linker flag too.