Clarification on clonefile / copyfile support of clone directories?

Question

Macho Man Randy Savage OP

Created 1w

Replies 5

Boosts 0

Participants 3

The man page of copyfile sates the following:

COPYFILE_CLONE [..] Note also that there is no support for cloning directories"

COPYFILE_CLONE_FORCE [...] Note also that there is no support for cloning directories: if a directory is provided as the source, an error will be returned.

Now the man page for clonefile:

> Cloning directories with these functions is strongly discouraged.  Use copyfile(3) to clone directories instead.

--

So am I to enumerate the content of a directory build subfolders along the way in the target destination and clone each file inside individually? If I recall NSFileManager seems to clone a large directory instantly (edit actually I remembered wrong NSFileManager does not do this. Finder seems to copy instead of clone as well).

On further inspection, clonefile states that it can do this, but it is discouraged. Interesting. I wonder why.

If src names a directory, the directory hierarchy is cloned as if each item was cloned individually. However, the use of clonefile(2) to clone directory hierarchies is strongly discouraged. Use copyfile(3) instead for copying directories.

P.S. - Forgive me if I posting this in the wrong category, I couldn't find a "category" in the list of available categories on these forums that seems appropriate for this question.

Answered by Systems Engineer in 879373022

That part of the copyfile man page is unfortunately somewhat poorly worded; further down it does clarify that:

 Note that recursive cloning is also supported with the COPYFILE_CLONE flag (but not the
 COPYFILE_CLONE_FORCE flag).  A recursive clone operation invokes copyfile() with
 COPYFILE_CLONE on every entry found in the source file-system object.  Because copyfile()
 does not allow the cloning of directories, a recursive clone will instead copy any directory
 it finds (while cloning its contents).  As symbolic links may point to directories, they are
 not followed during recursive clones even if the source is a symbolic link.  Additionally,
 because the COPYFILE_CLONE flag implies the COPYFILE_EXCL flag, recursive clones require a
 nonexistent destination.

So yes, you can use copyfile(3) to recursively clone directories. You should not use clonefile(2) for that purpose.

Boost

Answer 1

Macho Man Randy Savage OP

1w

I just found that Quinn wrote about this a bit here: https://developer.apple.com/forums/thread/784446

I'm still unsure why the man page for one function directs you to use another function - but that man page says cloning directories is unsupported so I'm still a bit confused about the recommendation. Is cloning directories unsupported or strongly discouraged?

Basically I have to move off NSFileManager because it doesn't provide progress and it doesn't support cancelling. But I'd like to clone when possible for speed and to save the user disk space.

0

Answer 2

Systems Engineer OP

Apple

1w

Accepted Answer

That part of the copyfile man page is unfortunately somewhat poorly worded; further down it does clarify that:

 Note that recursive cloning is also supported with the COPYFILE_CLONE flag (but not the
 COPYFILE_CLONE_FORCE flag).  A recursive clone operation invokes copyfile() with
 COPYFILE_CLONE on every entry found in the source file-system object.  Because copyfile()
 does not allow the cloning of directories, a recursive clone will instead copy any directory
 it finds (while cloning its contents).  As symbolic links may point to directories, they are
 not followed during recursive clones even if the source is a symbolic link.  Additionally,
 because the COPYFILE_CLONE flag implies the COPYFILE_EXCL flag, recursive clones require a
 nonexistent destination.

So yes, you can use copyfile(3) to recursively clone directories. You should not use clonefile(2) for that purpose.

1

Answer 3

Macho Man Randy Savage OP

1w

Thanks a lot for explaining that.

It's good to know that I can do this but I'm still hesitant to use it given the 'strongly discouraged' language in the documentation.

Does the 'strongly discouraged' stuff only apply to clonefile function or should I avoid 'cloning' directories in general with the copyfile function?

0

Answer 4

Systems Engineer OP

Apple

1w

The ‘strongly discouraged’ guidance only applies to directly using the clonefile function on directories, because it ties up the filesystem in ways that can be detrimental to the overall system health. On the other hand cloning directories recursively with copyfile is perfectly fine, because copyfile internally only calls clonefile on each file separately. Hope that helps clarify things.

1

Answer 5

DTS Engineer OP

Apple

5d

So, let me start here:

Should I avoid 'cloning' directories in general with the copyfile function?

No, quite the opposite. If you read the copyfile man page closely, you'll find that it's very careful to NEVER say that it "clones directories". It either refers to cloning individual files or "recursive cloning".

Recursive cloning means using COPYFILE_CLONE on directories and is actually the "standard" behavior across all of our copy APIs (and the Finder). More specifically, COPYFILE_CLONE means that copy file calls clonefile for every possible individual file object and creates new objects for directories or anywhere it can't use clonefile. Note that this process is more sophisticated than it might seem, as it doesn't just use cloning within the same file system, but actually preserves clones across volumes (which is much trickier).

copyfile() can also do standard "copy all data" copies on APFS but, to be honest, I've never come up with any good reason why you would do so, as it's basically the same as saying "please make this copy really large and extra slow".

That leads to here:

Does the 'strongly discouraged' stuff only apply to the clonefile function?

Yes, as that's the only API in the system that will actually clone a directory. However, the language in the clonefile man page is probably stronger and certainly less detailed than it should be. Earlier you mentioned a thread I wrote about this, and the summary of that thread is:

Calling clonefile on a directory blocks all modifications to the directory while the operation is in progress, which can be a huge problem if you call it on the "wrong" directory. Don't use it as a generic replacement for copyfile.
Directory cloning can be much faster (~10x) than copyfile(COPYFILE_CLONE) and that's a big enough performance gain that it might be worth using.

The key here is knowing how many files you're copying and where the files are located in the hierarchy. If your app controls those factors, then using clonefile to clone directories can be very useful.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

1