I can confirm that after rebooting the client system, I get the same error report from Terminal on the client system. Mac-mini:testSystem alan$ ll -R VAquaManager.app.1734032083499 total 32 drwxr-xr-x@ 1 alan staff 16384 Dec 12 11:34 Contents VAquaManager.app.1734032083499/Contents: total 64 drwxr-xr-x@ 1 alan staff 16384 Dec 8 09:37 runtime-arm drwxr-xr-x@ 1 alan staff 16384 Dec 8 09:37 runtime-x86 VAquaManager.app.1734032083499/Contents/runtime-arm: total 32 drwxr-xr-x 1 alan staff 16384 Dec 12 11:34 Contents VAquaManager.app.1734032083499/Contents/runtime-arm/Contents: total 0 ls: fts_read: Permission denied Using Terminal on the server system, I get: alan@Alans-iMac testSystem % ll -R VAquaManager.app.1734032083499 total 0 drwxr-xr-x@ 4 alan staff 136 Dec 12 11:34 Contents VAquaManager.app.1734032083499/Contents: total 0 drwxr-xr-x@ 3 alan staff 102 Dec 8 09:37 runtime-arm drwxr-xr-x@ 3 alan staff 102 Dec 8 09:37 runtime-x86 VAquaManager.app.1734032083499/Contents/runtime-arm: total 0 drwxr-xr-x@ 2 alan staff 68 Dec 12 11:34 Contents VAquaManager.app.1734032083499/Contents/runtime-arm/Contents: total 0 VAquaManager.app.1734032083499/Contents/runtime-x86: total 0 drwxr-xr-x@ 3 alan staff 102 Dec 12 11:35 Contents VAquaManager.app.1734032083499/Contents/runtime-x86/Contents: total 0 drwxr-xr-x@ 2 alan staff 68 Dec 12 11:35 Home VAquaManager.app.1734032083499/Contents/runtime-x86/Contents/Home: total 0

To clarify, I don't think APFS itself is a key factor. How long is your current retry timing? Certainly comparing local and network volumes, much longer timeouts are often required for a network file system. My timeout is currently 20 seconds, but I have previously used 60 seconds. I have never had a similar problem with local directory deletion. When local directories fail to delete, it is because of a spontaneously created .DS_Store file. What does "viewed" here actually mean? For example, using Terminal on the client system to list the directory, delete the directory, or list the parent directory. What's the file system on the SSD? APFS, HFS+, or something else? HFS+. I use this volume with various old macOS releases back to 10.10. On the delete side... what's the specific command/function your executing? When diagnosing from Terminal, rmdir. Probably the same system call from my application. As I mentioned above, I would not assume that the server is the failure point here. I believe the problem survives a client reboot. I will double check that after I submit this reply. Also, is the Finder actively open and monitoring these directories while all this is happening? The Finder doesn't normally create .DS_Store file for directories it isn't actually interacting with. Yes (on the server system). And I have observed .DS_Store files after the deletion attempt that were not there before the deletion attempt. (The Finder is open because after updating the applications on the server system, the next thing I will do is run some of them.) One more thing: I successfully implemented a workaround where the client moves the application to the (remote) trash instead of deleting it. That worked when the server system was 14.7.1, but fails after updating it to 14.7.2 (Operation not permitted). My only current workaround is to rename the application (within the same directory) and leave it there for me to delete manually using the Finder on the server system.

First off, any file system is inherently shared data structure that many process can modify at anytime. Strictly speaking, it isn't possible to create a program that is GUARANTEED to be able to delete any given directory- file deletion is generally slower than creation, so it's entirely possible for a process to create files faster than you can create them, preventing deletion. Of course, but that is not what is happening here. At most one file is being created, a .DS_Store file. Having encountered this problem before in a strictly local context, I changed my program to retry and that has worked (with local files). If the problem is APFS delaying the deletion of the directory, then perhaps I need a longer timeout on my retry loop? The real problem is not the failure to delete the directory, it is inconsistent state that is created (as viewed by a network client). Apparently, the file server believes the directory exists (it shows up when remotely listing the parent) even after the directory ceases to exist (as viewed on the server file system). At the very least, when the network client tries to list or delete this directory, the file server should notice that the directory no longer exists, update its state and return a nonexistent file error. Instead, it returns a permissions error but does not update its state, so any future attempt to list or delete the directory will get the same error. This bad state apparently lasted for days. I consider this a bug in the file server. I probably will modify my program to move the application before trying to delete it. I don't like the idea of adding macOS specific code to an application that is currently not OS specific, but that may be the only option. I believe that macOS specific code is needed to find a location (trash) where the Finder (or APFS) will not continue to try to calculate the application size. (Does moving the application to the trash stop APFS from recalculating its size?) The directories are not in the home directory. They are on an external SSD volume.

Ordinary Unix calls...

As I already mentioned, the client and server are both macOS systems and the server volume is offered using macOS file sharing. The connection is a few feet of ethernet. There are no multiple shares. As you suggest, the persistence seems to be on the server side. Not the file system, but the network server. Using Unix tools on the server to view the directories produces the expected results, except for the delay. Using Unix tools on the client to view or delete the directory fails. This failure still occurs even now. As I mentioned, the client is using ordinary Unix system class to delete files and directories. I just noticed that in some other applications that were installed by my program there is a .DS_Store file in the corresponding location. I do not have Calculate All Sizes enabled on the server system, but I notice that Finder displays the sizes of bundled applications. I suspect there is a problem with Finder trying to recalculate the size of the application concurrently with my program trying to delete the application. The problem might involve creating a .DS_Store file (a known problem) in D or it might just introduce a delay in the actual deletion of S. My program already has retry logic to handle the spontaneous creation of .DS_Store files during a directory tree deletion, but it cannot handle the permanent failure introduced by file sharing.

Kevin, I thank you for your response. I believe progress is happening. For clarity, let me call the arm machine the client system and the x86 machine the server system. I am deleting a directory tree on the server system from a Java application running on the client system. Java uses basic system calls (rmdir and unlink) to delete items. I put a breakpoint on the exception handler and discovered an interesting situation. The failure on directory deletion is directory not empty. That should not happen because before attempting to delete the directory, my program deleted its contents. When I examine the directory that I could not delete (D) in Terminal on the server system, it is indeed not empty. It contains an empty subdirectory (S), which my program previously "deleted". A few seconds later, directory S disappeared (as viewed in Terminal on the server system)! It appears that there is a race condition. The operation to delete S apparently succeeded, but did not take effect immediately. The operation to delete D somehow overtook the previous operation and failed as a result. From Terminal on the client system, S appears to exist but trying to list its contents fails with the fts_read error. I get the same error if I open a new Terminal window and navigate to D and try to list S. If I unmount the volume and reconnect, I see the same bad state in Terminal. Listing D shows S. Listing S gets the fts_read error. Is this a bug or am I doing something wrong? Is there a reliable way to work around this problem?

I can confirm that the problem I was having has been fixed in Xcode 16.1 and Command Line Tools 16.1. I was confused before because although Xcode 16.1 was installed, I was actually using Command Line Tools 16.0. You can tell if the library is correctly built if otool -l shows a hard link to the Quartz framework and no link to the QuickLookUI framework.

A very crude and heavy-handed workaround: Add a weak link to QuickLookUI as I described previously. Run the executable with DYLD_FORCE_FLAT_NAMESPACE=1. The essence of the problem is that my library compiled with recent releases of Xcode identifies QLPreviewView as belonging to the namespace assocated with QuickLookUI, but on macoS 11 and earlier, it is in the namespace associated with Quartz, so it is not found using the default two-level namespace.

Alas, I spoke too soon. Although adding the weak link to QuickLookUI allows the library to load on macOS 11, it does not link properly. The code fails trying to create a QLPreviewView.

I am using command line tools to build a dynamic library that uses QLPreviewView and targets macOS 10.10. The API predates 10.10 (in the Quartz framework) so there should be no problem. However, even after installing Xcode 16.1, the library contains a hard link to the QuickLookUI framework, causing it to fail on macOS releases prior to macOS 12. I assume this hard link is created because in the current SDK Quartz imports QuickLookUI. I presume that the link should be a weak link. A simple test program: #import <Quartz/Quartz.h> NSView *test() { NSRect bounds = NSMakeRect(0, 0, 1, 1); QLPreviewView *preview = [[QLPreviewView alloc] initWithFrame:bounds style:QLPreviewViewStyleCompact]; return preview; } The build script: cc -target x86_64-apple-macos10.10 -dynamiclib -ObjC -framework Quartz test.m The relevant output from otool: Load command 9 cmd LC_LOAD_DYLIB cmdsize 88 name /System/Library/Frameworks/Quartz.framework/Versions/A/Quartz (offset 24) time stamp 2 Wed Dec 31 16:00:02 1969 current version 1.0.0 compatibility version 1.0.0 Load command 11 cmd LC_LOAD_DYLIB cmdsize 96 name /System/Library/Frameworks/QuickLookUI.framework/Versions/A/QuickLookUI (offset 24) time stamp 2 Wed Dec 31 16:00:02 1969 current version 0.0.0 compatibility version 1.0.0 I was able to generate a weak link as follows: cc -target x86_64-apple-macos10.10 -dynamiclib -ObjC -weak_framework QuickLookUI -framework Quartz test.m I have confirmed that this fix produces a library that loads on macOS 11.

In general, you shouldn’t ship the .tbd file in your final product. I'm not sure what you mean by final product. In this case, my final product is a framework, and the immediate "customers" for that product are developers. The developers would be building things that use the framework (compile time role) and/or including it in applications (run time role). So, if I combine various things you have written, it sounds like a reasonable choice is for me to imitiate what the current Xcode does when it builds a framework. That means creating a V4 .tbd file with no UUIDs. Yes?

I like that answer! My explanation is a bit complicated. As you may know, the JavaNativeFoundation framework does not run natively on arm64. So, I am trying to figure out how to compile the open-source for it in a way that can be used on all architectures and macOS releases back to 10.10. I found that the only way to compile a framework to support 10.10 was to use an old version of Xcode (13.4.1 worked). When I build the framework using that Xcode, it creates a .tbd that includes UUIDs. Instead of using an old Xcode, I thought I would instead create the package using individual command tools (such as tapi) from the current Xcode. I figured that if I was unable to generate equivalent files, I must be doing something wrong. So, I'm happy to hear that the lack of UUIDs in the .tbd is not a problem. Now, you may be thinking that I should not need a JNF framework that runs on 10.10, as 10.10 already has a JNF. That seems reasonable, assuming that the dynamic linker ignores frameworks on the path that don't support the current OS and arch. However, someday I may wish to replace JNF with my own, much smaller framework, and I want that framework to run on 10.10 and later. So, my questions are: Do I need a .tbd file in a framework to run on some releases of macOS? If so, which version of the .tbd file should I use for maximum compatibility with older macOS release? Thank you for your assistance!

The cluprit turns out to be Finder! Yes, if a directory is nested below a directory that is displayed by Finder with the Calculate All Sizes view option enabled, Finder will cache the directory cumulative size in a .DS_Store file in that directory and will update the cached size in response to FSEvents. The upshot: if a user uses the Calculate All Sizes option, standard Unix commands such as /bin/rm -rf may fail intermittently.

FB10023961

Deleting a file (any file) may generate an FSEvent that triggers a background process to examine the directory or directory tree. If the background process somehow creates .DS_Store files, that would explain the behavior. Spotlight indexing and Time Machine backup come to mind as possible culprits.