How to improve CloudKit server latency when uploading data

I am having a hard time uploading data to my CloudKit container in a series of operations. I have an 'uploader' function in my app that can populate the CloudKit private database with a lot of user data. I batch the records into multiple CKModifyRecordsOperations (with 300 records in each operation as a maximum), before I upload them. When I do this with a lot of data (less than 50MB even), I get CKErrorRequestRateLimited errors for some of the operations, with a value for CKErrorRetryAfterKey specified in the CKError. I respond by creating new operations for the same records, delaying them by the specified CKErrorRetryAfterKey key, and then retrying the operations. But those operations end up getting more CKErrorRequestRateLimited errors again. Usually one or two operations go through at a time, and all others have the same error, and then I have to retry again after 90 seconds or whatever the retry-key says.  Doing all of this can take dozens of minutes to do a simple upload.

I checked the CloudKit dashboard, and for the container's 'developer' telemetry section, the 'server latency' seems very very high (over 100,000 for 95% percentile). It also suggests the 'average request size' is 150KB on average across the last few days as I've been testing this, which doesn't seem like a lot.

I've tried throttling the requests so only 20 modify operations are sent at a time, but it doesn't seem to help. I'm not sure what else I can try to improve this. I have 'query' indexes for 'recordName' field for each recordType, and 'query, searchable, sortable' on some of the custom fields on the recordTypes (though not all). The CKModifyRecordsOperations' configurations have 'qualityOfService' set to 'userInitiated'.

Is there anything else I can try to improve the upload speeds? Or is it out of my control?



I've now checked the Logs for the past hour or so, and it confirms that many of the requests have a size of 50KB, and the 'server latency' can be 10-11 seconds! This is with batching only 100 records per operation. I really need to improve on this, but not sure how, since it might be a server issue.
tnx
How to improve CloudKit server latency when uploading data
 
 
Q