After observing the system in more detail, I see more of what's happening. When I initiate an integration for a bot, a unix-executable file is generated in the folder /Library/Developer/XcodeServer/IntegrationAssets. This file is supposed to have the permissions -rwxrwx--- but instead has permissions -rw-r-r--.
Before: -rw-r--r-- 1 _xcsd _xcs 48948 Oct 9 14:12 5b6890ca4afc71b3c85c5f931176d970
After: -rwxrwx--- 1 _xcsd _xcs 48948 Oct 9 14:12 5b6890ca4afc71b3c85c5f931176d970
When the bot finishes and the file permissions are broken, the bot fails and gives me the warning about not being able to upload the integration assets. It also fails to delete the folder for the integration assets located at ~/Library/Caches/XCSBuilder/Integration-04f7a1d672eaf280a992b4001d5367b4 (the guid changes per build)
If I run sudo xcrun xcscontrol --fix-permissions in time, the permissions get fixed and the bot finishes normally.
Here's the real bummer: The permissions are only wrong about 1 out of every 4 integrations.
Anybody have any further suggestions? What script is creating that file with the incorrect permissions sometimes?