@Wittmason with "shadow baking" I meant "light baking". What I really mean is that in the Blender renderer (the screenshots at the bottom) you can see beautiful shadows. They are not generated when running on the Vision Pro.
An IBL will "tint" our assets' colour, but it will not add shadows. I think the only solution to have shadows is to replace the textures with textures that have those shadows baked in. I guess they call this "light baking" in Unity/Unreal, but Blender doesn't have a good workflow for this.
My current understanding:
an asset texture as a JPG is sometimes a smaller image used in repeated pattern on an asset
other times an asset texture is just colour/texture data, not using any jpg
In consideration of the above, what I understand the limitations of light baking to be:
When baking light onto a JPG texture (so you can see a shadow as part of that texture), then you basically need one big texture jpg per asset and cannot use any "repeated pattern" any more.
When baking light onto a texture that doesn't use any JPG but instead colour/texture data, then a newly JPG will need to be generated for that.
The above 1 & 2 makes it so the usdz file size will grow tremendously because of the new and large sizes texture JPGs...
I believe that Unity/Unreal therefore don't "bake light onto a texture" but instead they create a "lightmap" which is another layer that can be added on top of an asset with data on which parts need to be lightened/darkened. I am guessing this is a more economical method to do so, but I have no idea how these lightmaps are supported by RealityKit / Vision Pro.