copyfile Sometimes Fails to copy .DS_Store when Copying a Folder But Does Not Report Usable Error

In the callback, I can see that the failure occurs on a .DS_Store file inside the folder. So for a .DS_Store, it is simple enough for me to just ignore the error and return COPYFILE_SKIP, but the somewhat more concerning issue here is that the true error reason is seemingly not reported? In the callback, if I read errno, it is 0. When copyfile returns, it returns -1 after I return COPYFILE_QUIT (and errno is 0), so I don't know what the error is or the appropriate way to handle it.

FYI, the code to copyfile is actually open-source, so you can fairly easily find all of the places where your callback would have been called by looking for "COPYFILE_ERR" in the source.

However:

For .DS_Store, just skipping seems reasonable, but when copying a folder, it may be appropriate to get the true failure reason.

...I'd ignore all .DS_Store errors and, potentially, just skip copying them all together. The system will generate the file anytime it needs/wants to, and it's very likely that it won't consider the copied file "valid" and will end up regenerating it anyway.

But checking the last path component of the source path seems like a hack way to handle errors.

The big thing I'd look at here actually is the stage, not the error, as I suspect the reason errno isn't set is that particular stage doesn't really have an "error".

If a file in the copying folder with important user data, I can't just silently skip it.

Theoretically, a user might choose to store their data in a file named ".DS_Store“; however, if the system encounters a ".DS_Store" file that it can't process, then I believe it will just destroy the file and create a new one. Skipping these files is kinder than what the system will already do.

It isn't clear to me how I should properly proceed in a situation where I can't get the actual reason for the failure.

I think the idea you're getting at here is that you should be able to point your copy engine at any point in the file system, have it do the "right" thing based only on the information it's provided by copyfile, not external knowledge (like the full source path).

That's a lovely idea, but for better or worse, macOS just does not work that way. There are a huge number of special cases (most of which are basically undocumented), and that's before external factors like EndpointSecurity (that API basically lets the client app arbitrarily "veto" critical syscalls) become involved. Most of the time, these don't matter because the user is working with "their" file (which is the simple case), but if you want to work with the "full" system... Well, I don't think there's any way to do without grappling with a whole bunch of edge cases.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

I think the idea you're getting at here is that you should be able to point your copy engine at any point in the file system, have it do the "right" thing based only on the information it's provided by copyfile, not external knowledge (like the full source path). [...] That's a lovely idea, but for better or worse, macOS just does not work that way. [...] but if you want to work with the "full" system... Well, I don't think there's any way to do without grappling with a whole bunch of edge cases.

Well I did not directly point copyfile at .DS_Store and tell it to do the thing. I started a recursive copy on a folder, which happens to have a .DS_Store file in it. And on macOS just about every folder the user views in Finder will have one of those hidden in there, unfortunately. The folder I was copying was on a NAS, has like 24 videos in it. I'm doing this intentional slow copy to test everything out. So the copy gets like 98% done and then it needs a diaper change on the .DS_Store file!

I'd ignore all .DS_Store errors and, potentially, just skip copying them all together.

I have no intention of including .DS_Store in the copy. I am ignoring it now. I'm not really sure if they should even exist and probably shouldn't passively travel in a copy (maybe with the one exception when an advanced user is viewing hidden files and actively puts it in the pasteboard). It's okay with me if I can't do the copy on DS_Store, I guess. I generally want my app to do what the user is trying to do as long as it can't hurt them. My main issue is it doesn't report the reason why I can't do the copy. In fact sometimes I can do the copy but similar to the issue in the other thread it seems that when the .DS_Store is on the NAS, copyfile fails and that may have something to do with it. But I can copy .DS_Store that are on the local drive (perhaps because of the clone option I dk).

And yes Finder can copy it, even from the NAS. So I guess if I really wanted to copy it I might be able to dust off that old Carbon File Manager (but I actually don't). Mainly my concern is cryptic failures even though I am primarily dealing with user facing files in reasonable user-facing locations. I'm not really trying to travel to the underworld.

In any case if my app doesn't have the right privileges to copy .DS_Store or whatever (which are littered all over the file system) a normal error, even one with no recovery options seems appropriate. I really want to know if it failed because of something else and not permissions (perhaps because the way the data is stored). If there is another file I come across that stores data in a similar way...and the user would reasonably WANT to copy that file, that's actually what I'm more concerned about than .DS_Store.

Mystery (partially) solved. I just took a sneak peak at the extended attributes of the .DS_Store file and some have a resource fork (probably the ones that were failing to copy I BET!). So appears to be the same failure we are talking about here: https://developer.apple.com/forums/thread/814076

.DS_Stores on the NAS's with resource forks.. maybe that will help you figure out leaky compression