I think your answer to the first question is wrong @DTS Engineer You only need to upload changed keys, not all keys, if this is not the first upload.

This is a real issue, if we only know what record IDs to send but don't know which batches have previously finished uploading (so we don't know which fields were changed during previous upload process), we have to re-upload all fields in nextRecordZoneChangeBatch again and again. Say we have a ImagePost, and user only edited the title a few times, and we have to re-upload every CKAsset associated with it. That could be hundreds of MBs for every record. Take CoreData as a concrete example. It is not that realistic and easy to track changed fields in CoreData item itself. A CoreData save may fail, or conflict with savings from other threads, the conflict solving logic is hard to write and error prone if we are to record changes in database item itself. In order to solve this, we may: We only mark it as having changes before a save, and only do full re-upload of an item if the having changes mark still exist after app launches. Or we can use history transactions records to figure out what fields changed, and if we use it, we have to keep all history transactions and never delete any. We need to know which batch finished, or we cannot use those methods.

The CKRecord we get from CKRecord(coder: unarchiver) is not a fully materialized one, it only contains necessary systemFields, so it cannot figure out and upload only changedKeys. I wonder want can I do to workaround this.

Follow up If the engine works serially and do not ask for next batch before user finished handleEvent, the problem is solve and I had a misunderstanding.

Thank you for your response. @DTS Engineer In the sample code, it works fine because populateRecord(_:) always uploads all keys of a record. However, in a production application, we often only want to upload changed keys to save network overhead and memory usage. To do that efficiently, we need to know which version of a record was last successfully uploaded, so we can figure out which keys have changed since then. For example, suppose we have a Core Data entity “Item” with the following property changes: let instance = Item("Item-1") Change A: instance.x is updated. Change B: instance.y is updated. Change C: instance.x and instance.z are updated. If Change C happens after I tell the sync engine about A and B (between 5 and 6, before those changes finish uploading), the engine will eventually call nextRecordZoneChangeBatch(_:) again for “Item-1.” At that point, I need to determine which fields are newly changed and only populate those. Without a batch identifier, we don’t know exactly which subset of keys got uploaded in the previous batch and which ones still need updating. This forces us to re-upload everything every time.

Suppose: Change A: instance.x is updated. (batch 1) Change B: instance.y is updated. (batch 2) handleEvent — but is it for batch 1, batch 2, or both? Change C: instance.x and instance.z are updated. (batch 3) If handleEvent is coalesced so that I only receive it once batch 1 and 2 are definitively finished, I know exactly which keys made it up to the server and can upload only [x, z], not [x, y, z] for batch 3. If handleEvent is NOT coalesced and if batch 2 eventually fails later, and I never find out which batch failed, then I wouldn’t know that key [y] actually needs re-uploading. This is why I need to identify which batch finished or failed.

@DTS Engineer Hi, do we have any updates?

@DTS Engineer I believe the state property should not be used, and should be removed. Right after calling add(pendingChanges:), but before that new state is actually persisted, the app could crash, run out of memory, be force-quit, encounter a kernel panic, or lose power. The pending changes are lost, making the engine unable to synchronize properly on the next launch. Database changes is like a stream of events, and I would like to track where we were, so we can pick up from where we left. The current design, however, makes us unable to do that. Because the engine may or may not combine changes of the same CKRecord, and in the handle event callback, we don't have any information about this. The record may be in pending state, and if I add more pending changes to the same record, it may be combined and send in one network request, so I only receive ONE callback for it. But the record may be already in sending state, and if I add more pending changes to the same record, it will make another network request, and I will receive TWO callbacks for it. So I cannot decide the current progress in the callback. When the app launches again, I cannot pick up from where I left off.