filecopy fails with errno 34 "Result too large" when copying from NAS

A user of my app reported that when my app copies files from a QNAP NAS to a folder on their Mac, they get the error "Result too large". When copying the same files from the Desktop, it works.

Clarifying what you mean here, are you saying that the Finder can handle the copy and while your app cannot? Or that your app is doing both copies it works to one destination (the Finder) and fails to another? Or something else?

I'm going to assume the Finder works and your app fails (that's what typically happens) but please correct that if I'm wrong.

I asked them to reproduce the issue with the sample code below and they confirmed that it reproduces. They contacted QNAP for support who in turn contacted me saying that they are not sure they can do anything about it, and asking if Apple can help.

Can someone at Apple say anything about this?

I spent some time looking into this and there does appear to be a newly reported issue (r.165759954) that sounds similar to this. In terms of next steps, I have a few suggestions:

1. What do you "know" about the file? Is it a particular file or all copies from that source? If it's a particular file, what makes it different than the other files? This can be difficult to investigate without direct access to the file system, but the basic answer here is that "something" about that file makes it different and finding that difference generally explains the entire problem. The issue I mentioned about is tied to very large xattr copying (particularly the resource fork), so it shouldn't be happening with most "normal" file content.

2. Try doing the same copy with cp. If cp works, then that opens up a new investigation path. "cp" is open source and, while the code is somewhat complex, it's straightforward enough that it's reasonably straightforward to compare the two implementations. This then becomes a matter of simply reworking your implementation to match cp's.

3. If cp fails, that shifts the problem out of your code and onto the file system side. It also means that, assuming that the Finder works fine, the problem is specific to the "copyfile" copy stack (which the Finder doesn't actually use). I'm not sure I'd really "suggest" it, but that also opens up another potential workaround, which would be to retry the failing copy with the Carbon File Manager, as described here. I can't guarantee that will work, but I strongly suspect it will.

Finally, if you haven't already, please file a bug on this and post the bug number back here.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Sorry for the confusion, and you assumed correctly that copying in the Finder works, but I actually meant something else. What I meant was this: a user of my app reported that when my app copies files from a QNAP NAS to a folder on their Mac, they get the error "Result too large". When the app copies the same files from the Desktop folder, it works. Copying the files with the Finder, regardless their source location, works as well.

From what the user told me, the issue only happens with some files. I actually asked them to narrow the issue down as much as possible and they found a folder with a single file, and copying that folder always causes that error. Interestingly, the destination file is actually created but has a much smaller size than the source: 250 KB instead of 34 MB.

The user also shared that file as a zip archive on Dropbox with me. I ran the ls -l@ command on it to see its extended attributes and their size, the output was:

	com.apple.FinderInfo	      32 
	com.apple.ResourceFork	     286 
	com.apple.quarantine	      57 

Could com.apple.ResourceFork be the culprit? It doesn't seem to be exceedingly large to me, but maybe zipping a file doesn't correctly preserve extended attributes. Should I ask the user to run this command on their Mac?

I also found this Stackoverflow post from 13 years ago which sounds like it could be the same issue, although they were trying to copy the extended attributes with xattr, not with filecopy: https://stackoverflow.com/questions/13533796/copy-mac-com-apple-resourcefork-extended-attribute-causing-argument-list-too-lo

I created FB21805212.

Sorry for the confusion, and you assumed correctly that copying in the Finder works, but I actually meant something else. What I meant was this: a user of my app reported that when my app copies files from a QNAP NAS to a folder on their Mac, they get the error "Result too large". When the app copies the same files from the Desktop folder, it works. Copying the files with the Finder, regardless of their source location, works as well.

Yes, that all makes sense. There are basically two different issues that work here:

1. There appears to be a bug in SMB involving the handling of larger xattr. The issue is specific to smb, which is why this doesn't affect local file copies.

2. The Finder uses its own copy engine, which bypasses the problem.

More specifically, the Finder is almost certainly using the "..namedfork/rsrc" accessor that StackOverflow post mentions.

Covering a few details:

Interestingly, the destination file is actually created but has a much smaller size than the source: 250 KB instead of 34 MB.

The initial file gets created and its "data" was copied, but then failed during the resource fork copy operation.

Could com.apple.ResourceFork be the culprit?

Sort of, and this is making me feel like an old man...

So, when HFS was originally introduced as part of System 2.1 in 1985, one of its innovations was the introduction of "dual fork" files. Any file could actually be made up of two different "forks". The "data fork", which is what contained a file’s normal content, and the "resource fork", which contained structure data accessed through a special API called the "resource manager".

For most of Classic MacOS's history, the resource fork and resource manager were a foundational part of how the system and apps operated. Most notably, apps heavily relied on the resource fork for storing their own "app data". The modern bundled app architecture was basically created to replace the resource manager.

Moving forward in time (1990s->2000s), two different things were basically happening at the same time:

(1)
There were widespread and very geeky debates about how awesome/awful multi-fork file systems were. The majority (Windows/Unix) argument was that "a file" consisted only of a single data stream and the Classic MacOS was crazy.

(2)
File system designers came decided that multi-fork file systems were a FABULOUS idea. The issue here is that, historically, file systems were basically designed and evolved like this:

• A group sits down and decides exactly what data belongs "in the file system".

• The file system designer goes and designs a set of on-disk structures that hold that data.

• The file system ships; everyone is happy.

• Sometime later, someone comes to the file system architect and says "I've got this great idea and all I need you to do is store this extra..."

• The designer finds some nook and/or cranny to store said data that doesn't break anything.

• This process repeats until the file system has become sufficiently twisted up that the designer is forced to start over.

File system designers are not fools, so, having fallen for this trick a few times, they refused to be fooled again... and created the "Extended Attribute" ("xattrs"). That is, they created a mechanism that allowed a file to be made up of multiple independent streams so that the higher-level system could attach whatever it wanted to the file without bothering the file system designer again.

And, yes, the timing of these two trends was EXACTLY as ironic as it sounds. That is, file system designers were widely implementing multi-fork file systems at EXACTLY the same time the rest of world was loudly yelling about how silly they were.

With all that context...

Could com.apple.ResourceFork be the culprit?

Yes, but only in an indirect sense. Most extended attributes are fairly small (bytes or kb), but the resource fork is the exception. I don't know what its formal limits really are; 20MB-50MB was pretty common, and the Resource Manager was robust enough that DTS had a semi-famous "The Resource Manager is not a Database" technote. In this case, that means that any problem with copying large xattrs is going to fail with the resource fork, since it tends to be the biggest.

Shifting to the current issue:

The user also shared that file as a zip archive on Dropbox with me. I ran the ls -l@ command on it to see its extended attributes and their size; the output was:

I don't know what the file itself actually was, but I suspect this was actually a file system compressed file. This thread has more background on this, but many years ago (~2012), we introduced a mechanism that allowed the file system to "invisibly" compress specific files, and that mechanism reused the resource fork.

That's also why:

From what the user told me, the issue only happens with some files.

File system compression is largely invisible to the user, though you can see signs of it if you know what you're looking for. For example, if you "Get Info" on a directory like "/System/Library/CoreServices”, you'll find that its logical size is BIGGER (~200 MB) than its physical size:

Size: 867,557,303 bytes (616.8 MB on disk) for 719 items

Those differences accumulate, so the size gap on /System/Library is typically a savings of ~12+ GB.

Should I ask the user to run this command on their Mac?

The big thing to test here is moving the file with "cp". If cp fails, then I'm pretty confident that the problem is the bug I mentioned before. You can also have them do a "Get Info" on the file, as any compressed file will have the same logical size discrepancy.

although they were trying to copy the extended attributes with xattr, not with filecopy:

If you look at its implementation, xattr is doing a fairly naive single call to fsetxattr() which tries to write everything in a single call. That works fine for most attributes (because they're tiny) but will fail with large attributes, particularly the resource fork. copyfile is doing the right thing here (which is why it works fine at other locations) but smb is not.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thanks for all the details and the funny anecdote. I'll ask the user to run the cp command, but they told me they have some other work to do at the moment and need a break. In the meantime, since you said it's a SMB issue, is there a way I can reproduce the "Result too large" myself?

• Setting a large resource fork like mentioned in that Stackoverflow post causes the mentioned error "Argument list too long".
• Using the Terminal command cat file > file/..namedfork/rsrc like suggested by the first answer in that post to set the file itself as a resource fork of itself works (listing the extended attributes with ls -l@ file shows com.apple.ResourceFork 34669586 for my 34 MB test file), but calling copyfile as in the code I posted in my first post still works, even when the destination is on a FAT volume connected via SMB.
• When calling fsetxattr as in the code below, I can only set a custom attribute named "asdf", but when using "com.apple.ResourceFork" the function doesn't seem to do anything.

let f = open(path, O_RDWR)
if f < 0 {
    throw NSError(domain: NSPOSIXErrorDomain, code: Int(errno))
}
var buf = UnsafeMutableRawPointer.allocate(byteCount: 4, alignment: 4)
var r = fsetxattr(f, "asdf", &buf, 4, 0, 0)
if r != 0 {
    throw NSError(domain: NSPOSIXErrorDomain, code: Int(errno))
}
r = close(f)
if r != 0 {
    throw NSError(domain: NSPOSIXErrorDomain, code: Int(errno))
}

Output of ls -l@ file:

	asdf	       4 
	com.apple.FinderInfo	      32 
	com.apple.ResourceFork	34669586 
	com.apple.quarantine	      57