If I turn off File Sharing on the server, turn it back on, and reconnect the client to the remote volume, the client is then able to list the remote directory without getting Permission Denied errors. I believe this result demonstrates the bad state is somewhere in the file sharing pipeline (most likely in the file sharing daemon). The check itself is fast enough that the cost of trying to maintain a somewhat current cache (just for empty) would higher than the performance gain. What I had in mind was caching directory contents to speed up operations like listing the directory. It would then make sense to check the cache before trying to delete a directory. An explanation is needed for why the files that fail are usually, but not always the same each time. I suspect the predictability results from the operation sequence generated by rm -rf, which is always the same, and the variability is the result of small timing differences for each run. When a client perform a sequence of operations on a remote volume, is it guaranteed that the operations are performed on the actual file system in the same order? I'm suspicious of delete operations, which only return a status. Does the entire pipeline block until the file system returns a status for a delete operation? The other possibility, which now seems remote, is that the client Finder is interfering in some way. I use the Finder to connect to the remote volume and immediately close the Finder window, but that might not be enough. Is there a way to connect to the remote volume from the command line?

Alas, no change running 14.7.4 on the server.

Is 14.7.1 when this specifically started happening? There was a significant issue with security scoped bookmark resolution that was introduced in 14.7.1 and was only resolved in 14.7.3. Interesting point. I don't believe I experienced this problem on older systems. I will upgrade to 14.7.4 and see if that makes a difference. Is the server machine logged into by multiple users? No. Does the volume(s) you're seeing this in have the "Ignore Ownership on this volume" checkbox checked? Yes for the external drive, no for the internal drive. (I tested both cases.) The broader point here is that IF reseting the server resolves the issue, then the underlying issue is actually with the server's state/configuration, NOT the actual file system data. I believe we are in violent agreement. I don't follow how you think this works The Finder is not involved. The client is a command line program running on the client machine. It is presumably talking to the file sharing daemon on the server machine (via the client OS). My thought was that the file sharing daemon itself might cache some data to improve performance. The requirement that a directory needs to empty is something individual file systems enforce, NOT the higher level system. I can imagine the file sharing daemon enforcing this requirement when using a cache (if it actually does use a cache).

To test the hypothesis that extended attributes are involved, I tried copying a bunch of files from /Library/Developer/CommandLineTools (not all of them). There were no extended attributes in the original files and none on the copied files. Deleting the copies produced errors. Oddly, trying to delete a seond time produced more errors. Mac-mini:test6 alan$ rm -rf C rm: C/usr/lib/sourcekitd.framework/Versions/A: Permission denied rm: C/usr/lib/sourcekitd.framework/Versions: Permission denied rm: C/usr/lib/sourcekitd.framework: Permission denied rm: C/usr/lib/swift/pm/llbuild: Permission denied rm: C/usr/lib/swift/pm: Permission denied rm: C/usr/lib/swift: Permission denied rm: C/usr/lib: Permission denied rm: C/usr: Permission denied rm: C/SDKs/MacOSX15.2.sdk/usr: Permission denied rm: C/SDKs/MacOSX15.2.sdk: Permission denied rm: C/SDKs: Permission denied rm: C: Permission denied Mac-mini:test6 alan$ rm -rf C rm: C/usr/include: Permission denied rm: C/usr/lib/clang: Permission denied rm: C/usr/lib/tapi: Permission denied rm: C/usr/lib/sourcekitd.framework/Versions/A/XPCServices: Permission denied rm: C/usr/lib/sourcekitd.framework/Versions/A: Permission denied rm: C/usr/lib/sourcekitd.framework/Versions: Permission denied rm: C/usr/lib/sourcekitd.framework: Permission denied rm: C/usr/lib/sourcekitdInProc.framework: Permission denied rm: C/usr/lib/swift/pm/llbuild/llbuild.framework: Permission denied rm: C/usr/lib/swift/pm/llbuild: Permission denied rm: C/usr/lib/swift/pm: Permission denied rm: C/usr/lib/swift: Permission denied rm: C/usr/lib: Permission denied rm: C/usr/share: Permission denied rm: C/usr: Permission denied rm: C/SDKs/MacOSX15.2.sdk/usr/include: Permission denied rm: C/SDKs/MacOSX15.2.sdk/usr: Permission denied rm: C/SDKs/MacOSX15.2.sdk: Permission denied rm: C/SDKs: Permission denied rm: C: Permission denied Mac-mini:test6 alan$ xattr -lr C xattr: [Errno 13] Permission denied: 'C/usr/include' xattr: [Errno 13] Permission denied: 'C/usr/lib/clang' xattr: [Errno 13] Permission denied: 'C/usr/lib/tapi' xattr: [Errno 13] Permission denied: 'C/usr/lib/sourcekitd.framework/Versions/A/XPCServices' xattr: [Errno 13] Permission denied: 'C/usr/lib/sourcekitdInProc.framework' xattr: [Errno 13] Permission denied: 'C/usr/lib/swift/pm/llbuild/llbuild.framework' xattr: [Errno 13] Permission denied: 'C/usr/share' xattr: [Errno 13] Permission denied: 'C/SDKs/MacOSX15.2.sdk/usr/include'

I repeated the experiment using the internal SSD (APFS) on the target machine instead of the external SSD (HFS+). The results were the same, except that xattr gets one more error: xattr: [Errno 13] Permission denied: './VAquaManager.app/Contents/runtime-x86/Contents/Home/conf' xattr: [Errno 13] Permission denied: './VAquaManager.app/Contents/runtime-arm/Contents/Home/legal' xattr: [Errno 13] Permission denied: './VAquaManager.app/Contents/runtime-arm/Contents/Home/conf' I suspect the difference is due to timing.

After restarting the client and reconnecting to the remote volume, ls and xattr continue to get Permission denied errors. After restarting the server and reconnecting to the remote volume, ls and xattr succeed without error. It seems that the file sharing server is working with cached data that does not match the file system. Incorrect cached data could explain why the original recursive delete failed. For example, if deleting the last file in a directory does not immediately update the cached data for the parent, then the subsequent attempt to delete the parent directory will fail because the directory appears to not be empty (based on the cached data).

cp also copies the MACL Mac-mini:test4 alan$ pwd /Volumes/Alan’s iMac._smb._tcp.local/test4 Mac-mini:test4 alan$ xattr -lr /Applications/apps/VAquaManager.app /Applications/apps/VAquaManager.app: com.apple.macl: Mac-mini:test4 alan$ cp -pR /Applications/apps/VAquaManager.app . Mac-mini:test4 alan$ xattr -lr VAquaManager.app VAquaManager.app: com.apple.macl: Deletion from the client still fails: Mac-mini:test4 alan$ rm -rf VAquaManager.app rm: VAquaManager.app/Contents/runtime-arm/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-arm/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-arm: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-x86: Permission denied rm: VAquaManager.app/Contents: Permission denied rm: VAquaManager.app: Permission denied Finder most definitely does cache sizes, in .DS_Store files. I discovered this a while ago, because it can cause rm -rf to fail locally. Yes, I did check that the extended attribute was gone after I removed it (on the server using Terminal).

I don't think a standard copy would have generated a MACL, however, you specifically did not ask for a standard copy. You used "ditto", which tries to create a more exact "duplicate" of the original object, even when those semantics restrict access that object. What would create a standard copy, preferably an operation that can be performed on the command line? More specifically, if it did generate a MACL, that MACL would have been scoped to the creating process, which should have given unrestricted access to the newly created object. I'm not sure how that would work if the the newly created object is on a remote file system. Is ditto creating the MACL or is the file sharing daemon creating it? Is the MACL scoped to the file sharing daemon or to the client Terminal process? ...the Finder does not have it's own mechanism for caching directory sizes, particularly to disk. APFS does do this, but that implementation is internal to APFS and not something that would be exposed through an xattr. But the target volume is not APFS, it is Mac OS Extended (Journaled). The known behavior of displaying cumulative sizes for bundled applications is presumably still implemented by the Finder? If I use xattr -cr on the server to remove the extended attributes, including the MACL, from the application on the server volume, the clinet still gets errors: Mac-mini:test2 alan$ xattr -lr V* xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-arm/Contents/Home/conf' xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-x86/Contents/Home/conf' rm -rf V* rm: VAquaManager.app/Contents/runtime-arm/Contents/Home/conf: Permission denied rm: VAquaManager.app/Contents/runtime-arm/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-arm/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-arm: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents/Home/conf: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-x86: Permission denied rm: VAquaManager.app/Contents: Permission denied rm: VAquaManager.app: Permission denied

I tried removing the extended attributes from the source app, but it did not solve the problem. I also rebooted the server and avoided opening a Finder window on the target directory, but the problem persists. The following shows actions performed on the client while connected to the remote volume. Mac-mini:test alan$ xattr -lr /Applications/apps/VAquaManager.app /Applications/apps/VAquaManager.app: com.apple.macl: Mac-mini:test alan$ ditto /Applications/apps/VAquaManager.app VAquaManager.app Mac-mini:test alan$ xattr -lr VAquaManager.app VAquaManager.app/Contents/app/.jpackage.xml: com.apple.FinderInfo: VAquaManager.app/Contents/runtime-arm/Contents/Home/.DS_Store: com.apple.FinderInfo: VAquaManager.app: com.apple.macl: Mac-mini:test alan$ rm -rf VAquaManager.app rm: VAquaManager.app/Contents/runtime-arm/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-arm/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-arm: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents/Home: Permission denied rm: VAquaManager.app/Contents/runtime-x86/Contents: Permission denied rm: VAquaManager.app/Contents/runtime-x86: Permission denied rm: VAquaManager.app/Contents: Permission denied rm: VAquaManager.app: Permission denied Mac-mini:test alan$ xattr -lr VAquaManager.app xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-arm/Contents/Home/conf' xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-x86/Contents/Home/conf' On the server, I can list VAquaManager.app with no error. There are no extended attributes on the files that were not deleted. It seems odd that FinderInfo extended attributes were present before I tried to delete the application. That makes me wonder if the Finder is calculating sizes even when the directory is not displayed. On the other hand, I do have the directory open in Finder on the client. Is it possible that the Finder on the client would attach extended attributes to remote files? I closed the FInder window on the client and repeated the test to a new target directory on the server. I get the same result. One thing I'm not sure if is when the Finder removes its FSEvents listeners.

The part that I don't understand is why an attribute designed to warn users of an untrusted application would have the side effect of preventing the application from being deleted (or at least making it more difficult). Granted that users would not delete the application but instead move it to the trash, but from a file sharing client apparently I no longer have that option. Is it the case that I am able to delete the application on the server because Terminal has full disk access, but I cannot delete it remotely because a file sharing client does not have similar permission?

Using Howard Oakley's xattred program, I examined the original application on the client. (I built this application on the client machine.) The program reports that every file and directory in the application has the com.apple.provenance attribute with identical values. More interesting is that the file Contents/runtime-x86/Contents/_CodeSignature/CodeResources has the com.apple.quarantine attribute, with the value 0083;673245b6;Safari;71F7C727-7831-4DBB-B9FA-13D14B18F09D. I'm guessing that is because I copied this directory from a downloaded JDK. The corresponding file under Contents/runtime-arm does not have this attribute. Presumably, the process by which I obtained this JDK runtime was different. Is this the cause of the problem? Should I delete the com.apple.quarantine attribute? What process should I use to extract a runtime from a JDK? It seems that files in a JDK routinely have this attribute. A JDK is not an application, however, so it never gets cleared?

The application is copied using ditto.

I tried to replicate the situation on the server, but the newly copied application files do not have xattrs, so I am unable to say what error might have been reported. The file that I described had a value of 01 02 00 F5 28 1A 84 40 15 BA C9 when I inspected it on the server. However, its current value is empty. Other files in that application had the same attribute and it seems they all now have empty values. I do not see any other attributes.

I'm not sure there is a real difference. The client may have gotten an error when it tried to read the extended attributes of runtime-arm/Contents. In any case, the extended attribute is com.apple.provenance.

I am somewhat surprised that moving the application to a directory that is not and has never been displayed by Finder (before trying to delete the application) does not fix the problem.