I managed to get direct app to camera extension communication working, but I'm not sure I'm doing it right. The gist is this:
You make a launch daemon which acts as a helper. The code lives in your app bundle, but the process started by launchd on your behalf will run as root. The helper makes a XPC listener on a Mach port with a global name, which can be found by your app and by the camera extension. You use the helper as a kind of mailbox to let the extension know where to find the app. This approach is mentioned at various places on this forum and elsewhere on the Internet, but never in great detail.
In your app, find and connect to the helper. This is similar to connecting to a bundled XPC service, only you use initWithMachServiceName instead of initWithServiceName. Then make an anonymous listener, get its endpoint, and send that endpoint as a parameter to your helper. Store it in the helper.
In your extension, find and connect to the helper, exactly the same way you do so from the app. Make an XPC call into the helper to fetch the endpoint which the app stored (in the reply block of your call). I don't know how to deal with synchronization issues here, I guess there is a slim chance that the extension could fetch the endpoint value at a time when it is in the process of being written by the app.
Now, in the extension, build a connection around the endpoint you just received, with initWithListenerEndpoint.
From the extension, make a call into your app - this will cause your app's listener:shouldAcceptNewConnection to be called and you have established a direct communication between the extension and the app, which of course can be used in both directions.
There are a myriad of small things you have to get right - matching protocols at each end, allowing the extension to break out of its sandbox to look up a Mach service name with a properly formatted com.apple.security.temporary-exception.mach-lookup.global-name entitlement, ensuring the name of the service is prefixed by one of the app groups, the helper bundle ID has to be the same as its Mach service name etc etc.
Also, I might be doing something wrong and making it all too complicated. The Camera Extension is nicely self-contained. All the app code has to do is ask to activate it. But the helper requires an installer, and it has to be at a known location so that the launchd plist can find it. Which restricts you to putting your app in /Applications, or to doing some heroics to write a plist with the correct absolute path to the bundle at run-time, and then somehow installing this newly-generated plist, all of which is very far from drag-and-drops simplicity.
Also, the camera extension isn't replaced when you install a new one. _cmiodalassistants will stubbornly hang onto an instantiation of the older version until you reboot. I have done a lot of rebooting.