With some help from libreswan, it seems a temporary workaround is constrain the vpn server to only one proposal for ike and esp. The issue happens when macOS rekeys, sending an incorrect order of proposals.
For example, if using libreswan, edit properties in /etc/ipsec.d/ikev2.conf
rekey=no
pfs=no
ike=AES_GCM_C_256-HMAC_SHA2_256-ECP_256
phase2alg=AES_GCM_C_256
Then in the VPN Profile, set
<key>EnablePFS</key>
<integer>0</integer>
<key>ChildSecurityAssociationParameters</key>
<dict>
<key>DiffieHellmanGroup</key>
<integer>19</integer>
<key>EncryptionAlgorithm</key>
<string>AES-256</string>
<key>IntegrityAlgorithm</key>
<string>SHA2-256</string>
<key>LifeTimeInMinutes</key>
<integer>1440</integer>
</dict>
<key>IKESecurityAssociationParameters</key>
<dict>
<key>DiffieHellmanGroup</key>
<integer>19</integer>
<key>EncryptionAlgorithm</key>
<string>AES-256</string>
<key>IntegrityAlgorithm</key>
<string>SHA2-256</string>
<key>LifeTimeInMinutes</key>
<integer>1440</integer>
</dict>