However, keep in mind that this is a problem in the other direction as well. That is, the client cannot delete all the files it knows about and then assume the directory is empty, as someone else can be creating files at the same time it's deleting them.
I don't believe that this particular problem can be blamed on interference from other known parties, such as Finder or MDS. Such interference would only temporarily prevent a directory being deleted. As far as I can tell, the inability to delete the directory persists until File Sharing is restarted.
(I'm not worried about rogue agents that perpetually create and delete files.)
Recall that after restarting FIle Sharing, the client regained the ability to list the directory tree. Just now, in this state where clients can list the directory tree, I made another attempt to delete the (remaining) tree, and it failed (but not at the same directory).
Mac-mini:test6 alan$ rm -rf V*
rm: VAquaManager.app/Contents: Permission denied
rm: VAquaManager.app: Permission denied
Mac-mini:test6 alan$ ll V*
total 32
drwxr-xr-x 1 alan staff 16384 Feb 28 12:02 Contents
Mac-mini:test6 alan$ ll -R
total 32
drwxr-xr-x@ 1 alan staff 16384 Jan 14 19:59 VAquaManager.app
./VAquaManager.app:
total 32
drwxr-xr-x 1 alan staff 16384 Feb 28 12:02 Contents
./VAquaManager.app/Contents:
total 32
drwxr-xr-x 1 alan staff 16384 Feb 28 12:02 runtime-arm
./VAquaManager.app/Contents/runtime-arm:
total 0
ls: fts_read: Permission denied
There are problems reading the directory, but apparently no problems reading the directory metadata. (I'm not sure if this is new, as I had not tried all of these commands before now.)
Mac-mini:test6 alan$ ls -l VAquaManager.app/Contents/runtime-arm
total 0
ls: fts_read: Permission denied
Mac-mini:test6 alan$ ls VAquaManager.app/Contents/runtime-arm
ls: fts_read: Permission denied
Mac-mini:test6 alan$ xattr -l VAquaManager.app/Contents/runtime-arm
Mac-mini:test6 alan$ stat VAquaManager.app/Contents/runtime-arm
905970052 275529348 drwxr-xr-x 1 alan staff 0 16384 "Feb 28 12:13:31 2025" "Feb 28 12:02:56 2025" "Feb 28 12:02:56 2025" "Jan 14 19:59:24 2025" 4096 32 0 VAquaManager.app/Contents/runtime-arm
Mac-mini:test6 alan$ ls -ld VAquaManager.app/Contents/runtime-arm
drwxr-xr-x 1 alan staff 16384 Feb 28 12:02 VAquaManager.app/Contents/runtime-arm
The evidence at this point, along with your explanations, is that the problem is bad state in the file sharing daemon. What kind of state could this be? You say that caching is not useful, but it sure seems like some caching is happening.
From a performance perspective there is a pretty ENORMOUS performance benefit to SMB "assuming" operations will succeed and proceeding forward with a command stream instead of individually waiting on every single action.
If this is known behavior, clients can be written to double check when needed (accepting the performance hit). However, I won't hold my breath waiting for a new command line argument to rm.
It remains to be seen if only trying to delete directories that are known to be empty avoids this problem.
It also doesn't explain what makes this particular directory different.
When trying to delete the last item a directory, returning success too soon could cause failure on the attempt to delete the directory. Perhaps something about these directories increases the odds of a delay in the deletion of the last item that is too large.