After posting the above I did some extra digging and found this thread: https://developer.apple.com/forums/thread/811481
Thanks again to @DTS Engineer (Quinn) — the pointer to "Fixing an untrusted code signing certificate" was exactly what was needed. Posting the resolution here in case anyone else hits this.
Root cause: the "Developer ID Installer" certificate on my machine had a custom Keychain trust override set to kSecTrustSettingsResultTrustAsRoot, applied across almost every policy (SSL, Code Signing, S/MIME, etc.), each with "Allowed Error: CSSMERR_TP_CERT_EXPIRED" attached. Best guess is that someone, at some point in the past, used Keychain Access's "Always Trust" on this certificate to make an unrelated warning go away, without realising what that setting actually does.
The effect: every local tool (codesign, pkgutil --check-signature, spctl -a -t install) reported the certificate and the resulting .pkg as completely valid, because the override makes the local machine blindly trust the certificate rather than actually validating the chain. Apple's notary service, however, performs its own independent validation that isn't affected by local Keychain overrides at all — so it correctly saw through this and rejected every submission with "The binary is not signed with a valid Developer ID certificate," no matter how the package was built or re-signed.
This is why re-signing with productsign directly (bypassing our third-party packaging tool), accepting a separately-pending Program License Agreement, and rebuilding the whole project from scratch all made no difference — none of those touched the actual cause. The minimal repro test suggested earlier in this thread (Application cert + zip = Accepted, Installer cert + pkg = Invalid, even with a completely trivial throwaway package) was what pointed at the certificate itself rather than anything in our build pipeline.
Fix: found via security dump-trust-settings, which showed the override explicitly. Removed it with:
security remove-trusted-cert /path/to/exported/installer-cert.pem
(exported from Keychain Access first, or the same can be done directly in Keychain Access: double-click the certificate, expand Trust, set the first dropdown to "Use System Defaults" and everything else to "No value specified.")
After that, a freshly re-signed .pkg notarized cleanly on the first attempt.
Thanks again for pointing at the right doc rather than us continuing to chase the chain-building warning literally.
Topic:
Code Signing
SubTopic:
Notarization