Alright I'm back and I figured it out. Here my findings as it relates to my use case of trying to get screenshots I have saved as PNG attachments during UI tests:
The .xcresult bundle uses a custom storage format for attachments and other test artifacts
Within the result bundle's Data subdirectory, Image data (and presumably other attachment data) is stored in files whose names appear to be base64 encoded keys prefixed with "data."
These files are compressed using zstd. You can confirm this by running the file command on them.
Direct decompression of these files using zstd reveals PNG image data. Hurray!
As mentioned before, the filenames which are prefixed with data contain base64-encoded identifiers.
The database.sqlite3 database which is present in the result bundle contains a table called Attachments which has rows for each attachment.
Those rows have columns with the filename you provided the attachment in your test, as well as the encoded key which is appended to "data" to form the filename.
Those column names are name and xcResultKitPayloadRefId respectively.
With a little scripting you can parse the file names from the Data subdirectory, decode each file into the correct image format using zstd, look up the correct name for the file in the database using the base64 key from the original file name, and rename the newly converted PNG file accordingly.
I created a script to do all of this provided the path to the .xcresult bundle and put in a public gist on GitHub: https://gist.github.com/mhk4g/a81b16b27bfcd38a5cfc21861f171e1a
The script assumes PNG but you should be able to easily adapt it for other image formats.
Best of luck to you!