This answer provided the final piece of the solution for me (which was the "set-key-partition-list" magic):
https://apple.stackexchange.com/a/285320
Specifically, my build script effectively does the following:
Set up the signing keys:
set -e
/usr/bin/security create-keychain -p $KC_PASSWORD packaging.keychain
/usr/bin/security set-keychain-settings packaging.keychain
/usr/bin/security unlock-keychain -p $KC_PASSWORD packaging.keychain
/usr/bin/security list-keychains -d user -s packaging.keychain $OTHER_KEYCHAINS_IF_ANY
/usr/bin/security import $KEY_DIR/dev-id-app.Certificates.p12 -A -k packaging.keychain -P $KEY_PASSWORD
/usr/bin/security import $KEY_DIR/dev-id-inst.Certificates.p12 -A -k packaging.keychain -P $KEY_PASSWORD
/usr/bin/security set-key-partition-list -S apple-tool:,apple: -k $KC_PASSWORD -t private packaging.keychain
/usr/bin/security list-keychains -d user
/usr/bin/security find-identity -v # this should list the two keys imported above
followed by the signing:
/usr/bin/codesign --force --sign "Developer ID Application: $DEV_NAME ($TEAM_ID)" --keychain packaging.keychain --deep --timestamp -o runtime -vvvv --entitlements gen.build/entitlements.plist gen.build/pkgroot/Applications/$APP_NAME.app/Contents/Resources/lib/$DYLIB_NAME.dylib
...(repeated for each *.so and *.dylib)...
/usr/bin/codesign --force --sign "Developer ID Application: $DEV_NAME ($TEAM_ID)" --keychain packaging.keychain --deep --timestamp -o runtime -vvvv --entitlements gen.build/entitlements.plist gen.build/pkgroot/Applications/$APP_NAME.app
/usr/bin/codesign --verify --deep --strict -vvvv gen.build/pkgroot/Applications/$APP_NAME.app # this is optional
using an entitlements.plist that might look something like this:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.app-sandbox</key>
<false/>
<key>com.apple.security.device.bluetooth</key>
<false/>
<key>com.apple.security.files.user-selected.read-write</key>
<true/>
<key>com.apple.security.network.client</key>
<true/>
<key>com.apple.security.get-task-allow</key>
<false/>
</dict>
</plist>
followed by the packaging (which also uses the keychain):
APP_BUNDLE_ID=com.example.my-app-name-lowercase # put something suitable here, e.g. com.example.my-app-name-lowercase
pkgbuild --identifier $APP_BUNDLE_ID --version $VSN_MAJOR.$VSN_MINOR.$VSN_BUILDNUM --root gen.build/pkgroot/Applications --component-plist gen.build/components.plist --install-location /Applications gen.build/$APP_NAME.pkg
using a components.plist file that might look like this (but plug in your app name in place of $APP_NAME):
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
<dict>
<key>BundleHasStrictIdentifier</key>
<true/>
<key>BundleIsRelocatable</key>
<false/>
<key>BundleIsVersionChecked</key>
<false/>
<key>BundleOverwriteAction</key>
<string>upgrade</string>
<key>RootRelativeBundlePath</key>
<string>$APP_NAME.app</string>
</dict>
</array>
</plist>
followed by building the outer package for the installer:
productbuild --sign "Developer ID Installer: $DEV_NAME ($TEAM_ID)" --keychain packaging.keychain --distribution gen.build/distribution.plist --package-path gen.build --resources $APP_NAME/Resources gen.build/$APP_NAME-$VSN_MAJOR.$VSN_MINOR.$VSN_BUILDNUM-Mac-installer.pkg
using a distribution.plist file that might look like this (but plug in your app's identifier in place of $APP_BUNDLE_ID, app name in place of $APP_NAME, and minimum macOS version (e.g. 10.8) in place of $MACOSX_DEPLOYMENT_TARGET):
<?xml version="1.0" encoding="utf-8"?>
<installer-gui-script minSpecVersion="2">
<title>$APP_NAME</title>
<pkg-ref id="$APP_BUNDLE_ID"/>
<options customize="never" require-scripts="false"/>
<volume-check>
<allowed-os-versions>
<os-version min="$MACOSX_DEPLOYMENT_TARGET"/>
</allowed-os-versions>
</volume-check>
<choices-outline>
<line choice="default">
<line choice="$APP_BUNDLE_ID"/>
</line>
</choices-outline>
<choice id="default"/>
<choice id="$APP_BUNDLE_ID" visible="false">
<pkg-ref id="$APP_BUNDLE_ID"/>
</choice>
<pkg-ref id="$APP_BUNDLE_ID">$APP_NAME.pkg</pkg-ref>
</installer-gui-script>
followed by telling the launch service that it should NOT remember that package location, to avoid weird and surprising behavior later:
/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -u `cd gen.build/pkgroot/Applications/$APP_NAME.app; /bin/pwd -P`
and then to submit for notarization:
xcrun altool --notarize-app --file gen.build/$APP_NAME-$VSN_MAJOR.$VSN_MINOR.$VSN_BUILDNUM-Mac-installer.pkg --username $NOTARIZATION_USERNAME --password $NOTARIZATION_PASSWORD --asc-provider $TEAM_ID --primary-bundle-id $APP_BUNDLE_ID
Note that the notarization password is an app-specific password specifically for notarizing (not specific to the app that you are notarizing, despite the name!), as described here:
https://stackoverflow.com/questions/56890749/macos-notarize-in-script
My script captures the RequestUUID (e.g. b8c31cd3-5f41-4adc-b062-da7eb65fc650) from the "xcrun altool" output, and then queries for the result every 10 seconds:
xcrun altool --notarization-info $REQUEST_UUID --username $NOTARIZATION_USERNAME --password $NOTARIZATION_PASSWORD --asc-provider $TEAM_ID
And then once it's done, it fetches and checks the log, then finally staples the notarization result to the package, so that it can be installed when not connected to the internet (since otherwise the installer checks with Apple's servers to see if the package has been notarized):
xcrun stapler staple gen.build/$APP_NAME-$VSN_MAJOR.$VSN_MINOR.$VSN_BUILDNUM-Mac-installer.pkg
The result is a Mac native .pkg file that can be downloaded and installed directly (by just double-clicking it in Finder) and cleanly, on everything from Mac OS X 10.7 Lion (and probably earlier) up to macOS 12.2+ Monterey, with none of that mount-the-DMG-and-drag-the-icon silliness. And the whole build process described above works while logged in via ssh - no GUI required.
