I tried to replicate the situation on the server, but the newly copied application files do not have xattrs, so I am unable to say what error might have been reported. The file that I described had a value of 01 02 00 F5 28 1A 84 40 15 BA C9 when I inspected it on the server. However, its current value is empty. Other files in that application had the same attribute and it seems they all now have empty values. I do not see any other attributes.

The application is copied using ditto.

Using Howard Oakley's xattred program, I examined the original application on the client. (I built this application on the client machine.) The program reports that every file and directory in the application has the com.apple.provenance attribute with identical values. More interesting is that the file Contents/runtime-x86/Contents/_CodeSignature/CodeResources has the com.apple.quarantine attribute, with the value 0083;673245b6;Safari;71F7C727-7831-4DBB-B9FA-13D14B18F09D. I'm guessing that is because I copied this directory from a downloaded JDK. The corresponding file under Contents/runtime-arm does not have this attribute. Presumably, the process by which I obtained this JDK runtime was different. Is this the cause of the problem? Should I delete the com.apple.quarantine attribute? What process should I use to extract a runtime from a JDK? It seems that files in a JDK routinely have this attribute. A JDK is not an application, however, so it never gets cleared?

The part that I don't understand is why an attribute designed to warn users of an untrusted application would have the side effect of preventing the application from being deleted (or at least making it more difficult). Granted that users would not delete the application but instead move it to the trash, but from a file sharing client apparently I no longer have that option. Is it the case that I am able to delete the application on the server because Terminal has full disk access, but I cannot delete it remotely because a file sharing client does not have similar permission?

I tried removing the extended attributes from the source app, but it did not solve the problem. I also rebooted the server and avoided opening a Finder window on the target directory, but the problem persists. The following shows actions performed on the client while connected to the remote volume. Mac-mini:test alan$ xattr -lr /Applications/apps/VAquaManager.app /Applications/apps/VAquaManager.app: com.apple.macl: Mac-mini:test alan$ ditto /Applications/apps/VAquaManager.app VAquaManager.app Mac-mini:test alan$ xattr -lr VAquaManager.app VAquaManager.app/Contents/app/.jpackage.xml: com.apple.FinderInfo: VAquaManager.app/Contents/runtime-arm/Contents/Home/.DS_Store: com.apple.FinderInfo: VAquaManager.app: com.apple.macl: Mac-mini:test alan$ rm -rf VAquaManager.app rm: VAquaManager.app/Contents/runtime-arm/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-arm/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-arm: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-x86: Permission denied rm: VAquaManager.app/Contents: Permission denied rm: VAquaManager.app: Permission denied Mac-mini:test alan$ xattr -lr VAquaManager.app xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-arm/Contents/Home/conf' xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-x86/Contents/Home/conf' On the server, I can list VAquaManager.app with no error. There are no extended attributes on the files that were not deleted. It seems odd that FinderInfo extended attributes were present before I tried to delete the application. That makes me wonder if the Finder is calculating sizes even when the directory is not displayed. On the other hand, I do have the directory open in Finder on the client. Is it possible that the Finder on the client would attach extended attributes to remote files? I closed the FInder window on the client and repeated the test to a new target directory on the server. I get the same result. One thing I'm not sure if is when the Finder removes its FSEvents listeners.

I don't think a standard copy would have generated a MACL, however, you specifically did not ask for a standard copy. You used "ditto", which tries to create a more exact "duplicate" of the original object, even when those semantics restrict access that object. What would create a standard copy, preferably an operation that can be performed on the command line? More specifically, if it did generate a MACL, that MACL would have been scoped to the creating process, which should have given unrestricted access to the newly created object. I'm not sure how that would work if the the newly created object is on a remote file system. Is ditto creating the MACL or is the file sharing daemon creating it? Is the MACL scoped to the file sharing daemon or to the client Terminal process? ...the Finder does not have it's own mechanism for caching directory sizes, particularly to disk. APFS does do this, but that implementation is internal to APFS and not something that would be exposed through an xattr. But the target volume is not APFS, it is Mac OS Extended (Journaled). The known behavior of displaying cumulative sizes for bundled applications is presumably still implemented by the Finder? If I use xattr -cr on the server to remove the extended attributes, including the MACL, from the application on the server volume, the clinet still gets errors: Mac-mini:test2 alan$ xattr -lr V* xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-arm/Contents/Home/conf' xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-x86/Contents/Home/conf' rm -rf V* rm: VAquaManager.app/Contents/runtime-arm/Contents/Home/conf: Permission denied rm: VAquaManager.app/Contents/runtime-arm/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-arm/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-arm: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents/Home/conf: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-x86: Permission denied rm: VAquaManager.app/Contents: Permission denied rm: VAquaManager.app: Permission denied

cp also copies the MACL Mac-mini:test4 alan$ pwd /Volumes/Alan’s iMac._smb._tcp.local/test4 Mac-mini:test4 alan$ xattr -lr /Applications/apps/VAquaManager.app /Applications/apps/VAquaManager.app: com.apple.macl: Mac-mini:test4 alan$ cp -pR /Applications/apps/VAquaManager.app . Mac-mini:test4 alan$ xattr -lr VAquaManager.app VAquaManager.app: com.apple.macl: Deletion from the client still fails: Mac-mini:test4 alan$ rm -rf VAquaManager.app rm: VAquaManager.app/Contents/runtime-arm/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-arm/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-arm: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-x86: Permission denied rm: VAquaManager.app/Contents: Permission denied rm: VAquaManager.app: Permission denied Finder most definitely does cache sizes, in .DS_Store files. I discovered this a while ago, because it can cause rm -rf to fail locally. Yes, I did check that the extended attribute was gone after I removed it (on the server using Terminal).

After restarting the client and reconnecting to the remote volume, ls and xattr continue to get Permission denied errors. After restarting the server and reconnecting to the remote volume, ls and xattr succeed without error. It seems that the file sharing server is working with cached data that does not match the file system. Incorrect cached data could explain why the original recursive delete failed. For example, if deleting the last file in a directory does not immediately update the cached data for the parent, then the subsequent attempt to delete the parent directory will fail because the directory appears to not be empty (based on the cached data).

I repeated the experiment using the internal SSD (APFS) on the target machine instead of the external SSD (HFS+). The results were the same, except that xattr gets one more error: xattr: [Errno 13] Permission denied: './VAquaManager.app/Contents/runtime-x86/Contents/Home/conf' xattr: [Errno 13] Permission denied: './VAquaManager.app/Contents/runtime-arm/Contents/Home/legal' xattr: [Errno 13] Permission denied: './VAquaManager.app/Contents/runtime-arm/Contents/Home/conf' I suspect the difference is due to timing.

To test the hypothesis that extended attributes are involved, I tried copying a bunch of files from /Library/Developer/CommandLineTools (not all of them). There were no extended attributes in the original files and none on the copied files. Deleting the copies produced errors. Oddly, trying to delete a seond time produced more errors. Mac-mini:test6 alan$ rm -rf C rm: C/usr/lib/sourcekitd.framework/Versions/A: Permission denied rm: C/usr/lib/sourcekitd.framework/Versions: Permission denied rm: C/usr/lib/sourcekitd.framework: Permission denied rm: C/usr/lib/swift/pm/llbuild: Permission denied rm: C/usr/lib/swift/pm: Permission denied rm: C/usr/lib/swift: Permission denied rm: C/usr/lib: Permission denied rm: C/usr: Permission denied rm: C/SDKs/MacOSX15.2.sdk/usr: Permission denied rm: C/SDKs/MacOSX15.2.sdk: Permission denied rm: C/SDKs: Permission denied rm: C: Permission denied Mac-mini:test6 alan$ rm -rf C rm: C/usr/include: Permission denied rm: C/usr/lib/clang: Permission denied rm: C/usr/lib/tapi: Permission denied rm: C/usr/lib/sourcekitd.framework/Versions/A/XPCServices: Permission denied rm: C/usr/lib/sourcekitd.framework/Versions/A: Permission denied rm: C/usr/lib/sourcekitd.framework/Versions: Permission denied rm: C/usr/lib/sourcekitd.framework: Permission denied rm: C/usr/lib/sourcekitdInProc.framework: Permission denied rm: C/usr/lib/swift/pm/llbuild/llbuild.framework: Permission denied rm: C/usr/lib/swift/pm/llbuild: Permission denied rm: C/usr/lib/swift/pm: Permission denied rm: C/usr/lib/swift: Permission denied rm: C/usr/lib: Permission denied rm: C/usr/share: Permission denied rm: C/usr: Permission denied rm: C/SDKs/MacOSX15.2.sdk/usr/include: Permission denied rm: C/SDKs/MacOSX15.2.sdk/usr: Permission denied rm: C/SDKs/MacOSX15.2.sdk: Permission denied rm: C/SDKs: Permission denied rm: C: Permission denied Mac-mini:test6 alan$ xattr -lr C xattr: [Errno 13] Permission denied: 'C/usr/include' xattr: [Errno 13] Permission denied: 'C/usr/lib/clang' xattr: [Errno 13] Permission denied: 'C/usr/lib/tapi' xattr: [Errno 13] Permission denied: 'C/usr/lib/sourcekitd.framework/Versions/A/XPCServices' xattr: [Errno 13] Permission denied: 'C/usr/lib/sourcekitdInProc.framework' xattr: [Errno 13] Permission denied: 'C/usr/lib/swift/pm/llbuild/llbuild.framework' xattr: [Errno 13] Permission denied: 'C/usr/share' xattr: [Errno 13] Permission denied: 'C/SDKs/MacOSX15.2.sdk/usr/include'

Is 14.7.1 when this specifically started happening? There was a significant issue with security scoped bookmark resolution that was introduced in 14.7.1 and was only resolved in 14.7.3. Interesting point. I don't believe I experienced this problem on older systems. I will upgrade to 14.7.4 and see if that makes a difference. Is the server machine logged into by multiple users? No. Does the volume(s) you're seeing this in have the "Ignore Ownership on this volume" checkbox checked? Yes for the external drive, no for the internal drive. (I tested both cases.) The broader point here is that IF reseting the server resolves the issue, then the underlying issue is actually with the server's state/configuration, NOT the actual file system data. I believe we are in violent agreement. I don't follow how you think this works The Finder is not involved. The client is a command line program running on the client machine. It is presumably talking to the file sharing daemon on the server machine (via the client OS). My thought was that the file sharing daemon itself might cache some data to improve performance. The requirement that a directory needs to empty is something individual file systems enforce, NOT the higher level system. I can imagine the file sharing daemon enforcing this requirement when using a cache (if it actually does use a cache).

Alas, no change running 14.7.4 on the server.

If I turn off File Sharing on the server, turn it back on, and reconnect the client to the remote volume, the client is then able to list the remote directory without getting Permission Denied errors. I believe this result demonstrates the bad state is somewhere in the file sharing pipeline (most likely in the file sharing daemon). The check itself is fast enough that the cost of trying to maintain a somewhat current cache (just for empty) would higher than the performance gain. What I had in mind was caching directory contents to speed up operations like listing the directory. It would then make sense to check the cache before trying to delete a directory. An explanation is needed for why the files that fail are usually, but not always the same each time. I suspect the predictability results from the operation sequence generated by rm -rf, which is always the same, and the variability is the result of small timing differences for each run. When a client perform a sequence of operations on a remote volume, is it guaranteed that the operations are performed on the actual file system in the same order? I'm suspicious of delete operations, which only return a status. Does the entire pipeline block until the file system returns a status for a delete operation? The other possibility, which now seems remote, is that the client Finder is interfering in some way. I use the Finder to connect to the remote volume and immediately close the Finder window, but that might not be enough. Is there a way to connect to the remote volume from the command line?

I just noticed something interesting in these (old) results: Mac-mini:test alan$ rm -rf VAquaManager.app rm: VAquaManager.app/Contents/runtime-arm/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-arm/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-arm: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-x86: Permission denied rm: VAquaManager.app/Contents: Permission denied rm: VAquaManager.app: Permission denied Mac-mini:test alan$ xattr -lr VAquaManager.app xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-arm/Contents/Home/conf' The file VAquaManager.app/Contents/runtime-arm/Contents/Home/conf was not deleted, but rm did not complain about it. It only complained about its parent, which could not be deleted because it was not empty. That tells me that the system call did not return an error, which makes me think that the file sharing daemon returned an OK status before it was known that the directory was not deleted. I observed this problem earlier when using my own program to delete. It is what made me think that operations were being performed out of order. Regarding Finder, the client Finder responds quickly when a change is made on the server. I assume it getting notifications of changes, which implies that the file sharing daemon is also getting notifications of changes. That could explain how caching done by the file sharing daemon could get out of sync with the file system. It could also be that the file system's implementation of FSEvents is causing the problem. When file sharing is turned off, the FSEvents listeners are unregistered and any associated state in the file system is presumably flushed. The problem is to explain why local FS operations do not fail. Does the file sharing daemon use alternative or private APIs to read directories?

However, keep in mind that this is a problem in the other direction as well. That is, the client cannot delete all the files it knows about and then assume the directory is empty, as someone else can be creating files at the same time it's deleting them. I don't believe that this particular problem can be blamed on interference from other known parties, such as Finder or MDS. Such interference would only temporarily prevent a directory being deleted. As far as I can tell, the inability to delete the directory persists until File Sharing is restarted. (I'm not worried about rogue agents that perpetually create and delete files.) Recall that after restarting FIle Sharing, the client regained the ability to list the directory tree. Just now, in this state where clients can list the directory tree, I made another attempt to delete the (remaining) tree, and it failed (but not at the same directory). Mac-mini:test6 alan$ rm -rf V* rm: VAquaManager.app/Contents: Permission denied rm: VAquaManager.app: Permission denied Mac-mini:test6 alan$ ll V* total 32 drwxr-xr-x 1 alan staff 16384 Feb 28 12:02 Contents Mac-mini:test6 alan$ ll -R total 32 drwxr-xr-x@ 1 alan staff 16384 Jan 14 19:59 VAquaManager.app ./VAquaManager.app: total 32 drwxr-xr-x 1 alan staff 16384 Feb 28 12:02 Contents ./VAquaManager.app/Contents: total 32 drwxr-xr-x 1 alan staff 16384 Feb 28 12:02 runtime-arm ./VAquaManager.app/Contents/runtime-arm: total 0 ls: fts_read: Permission denied There are problems reading the directory, but apparently no problems reading the directory metadata. (I'm not sure if this is new, as I had not tried all of these commands before now.) Mac-mini:test6 alan$ ls -l VAquaManager.app/Contents/runtime-arm total 0 ls: fts_read: Permission denied Mac-mini:test6 alan$ ls VAquaManager.app/Contents/runtime-arm ls: fts_read: Permission denied Mac-mini:test6 alan$ xattr -l VAquaManager.app/Contents/runtime-arm Mac-mini:test6 alan$ stat VAquaManager.app/Contents/runtime-arm 905970052 275529348 drwxr-xr-x 1 alan staff 0 16384 "Feb 28 12:13:31 2025" "Feb 28 12:02:56 2025" "Feb 28 12:02:56 2025" "Jan 14 19:59:24 2025" 4096 32 0 VAquaManager.app/Contents/runtime-arm Mac-mini:test6 alan$ ls -ld VAquaManager.app/Contents/runtime-arm drwxr-xr-x 1 alan staff 16384 Feb 28 12:02 VAquaManager.app/Contents/runtime-arm The evidence at this point, along with your explanations, is that the problem is bad state in the file sharing daemon. What kind of state could this be? You say that caching is not useful, but it sure seems like some caching is happening. From a performance perspective there is a pretty ENORMOUS performance benefit to SMB "assuming" operations will succeed and proceeding forward with a command stream instead of individually waiting on every single action. If this is known behavior, clients can be written to double check when needed (accepting the performance hit). However, I won't hold my breath waiting for a new command line argument to rm. It remains to be seen if only trying to delete directories that are known to be empty avoids this problem. It also doesn't explain what makes this particular directory different. When trying to delete the last item a directory, returning success too soon could cause failure on the attempt to delete the directory. Perhaps something about these directories increases the odds of a delay in the deletion of the last item that is too large.