Thanks for your response. I've downloaded and extracted Kernel_Debug_Kit_15.4_build_24E248.dmg. Here's what I've tried so far to sybolicate the sample kernel panic log. Note: the kernel passed in to the -o flag of atos(1) is the one that is found in kernel.release.t8112.dSYM/Contents/Resources/DWARF/kernel.release.t8112. The load address computed as the kernel base addresses given in the kernel panic log minus the corresponding slide atos -o kernel.release.t8112 -arch arm64e -l FFFFFE0007004000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 No output generated The value of Kernel slide given in the kernel panic log passed to the -s flag atos -o kernel.release.t8112 -arch arm64e -s 0000000014610000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 No output generated The value of KernelCache slide given in the kernel panic log passed to the -s flag atos -o kernel.release.t8112 -arch arm64e -s 0000000014608000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 No output generated The value of Kernel text exec slide given in the kernel panic log passed to the -s flag atos -o kernel.release.t8112 -arch arm64e -s 0000000015D88000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 Output: Looks bogus __doprnt.digits (in kernel.release.t8112) + 103398 bootstrap_pagetables (in kernel.release.t8112) + 329876 bootstrap_pagetables (in kernel.release.t8112) + 322192 table_10bits (in kernel.release.t8112) + 10400 __doprnt.digits (in kernel.release.t8112) + 101598 recvmsg_x (in kernel.release.t8112) + 332 IOMachPort::noMoreSendersForObject(OSObject*, unsigned int, unsigned int*) (in kernel.release.t8112) + 440 IOUserClient::init(OSDictionary*) (in kernel.release.t8112) + 0 IOServiceMessageUserNotification::MetaClass::~MetaClass() (in kernel.release.t8112) + 528 bootstrap_pagetables (in kernel.release.t8112) + 336156 table_10bits (in kernel.release.t8112) + 10520 ropagetable_end (in kernel.release.t8112) + 2688 ropagetable_end (in kernel.release.t8112) + 2688 ipsec_cleanup_family (in kernel.release.t8112) + 920 pf_test (in kernel.release.t8112) + 3700 ropagetable_begin (in kernel.release.t8112) + 324884 nfU16InvMasks (in kernel.release.t8112) + 112 Can you please instruct me how to symbolicate my sample kernel panic log properly.

That sounds reasonable. Thanks a lot for your really valuable input.

Thanks for all your help.

Thanks very much for your response. I did manage to get it to work by purging the local name cache as well as the hash table where FSNodes are being kept in vnop_getattr() on receiving a notification from the server of a directory entry no longer existing. I do appreciate that it's not an ideal solution though for the reasons you described. Apple have made VNOP_MONITOR() available as a public symbol in macOS 15.4 beta in response to the bug report I submitted, FB16082106. Is there anywhere I could look for VNOP_MONITOR()+vnode_notify() usage examples other than the SMBClient and Darwin/XNU sources as it's not immediately clear how remote filesystem changes are propagated across all connected clients through the use of those functions?

With VNOP_MONITOR() still unavailable I thought I'd explore other alternatives. Passing VFS_TBLNOMACLABEL into vfs_fsadd() has revealed that the issue was with the vnode being NULL when accessing directories that were removed on other client machines. Further debugging has confirmed that the vnode was being set to NULL via a call to cache_enter() with cn_flags set to MAKEENTRY in vnop_lookup() on receiving a response from the server saying that the directory was no longer there. Dtrace(1)'ing vnode's mac labels showed that for my filesystem they were set to NULL at all times whether or not VFS_TBLNOMACLABEL was set. My filesystem being distributed, I have the benefit of querying the server about the presence of files or directories in my vnop_lookup(). What I attempted next is modify my lookup algorithm to remove the local vnode that represents a file or directory that the server reports as no longer existing, i.e. removed by another client machine. Below is a pseudocode for the part of my lookup algorithm that deals with negative server responses. ret = query_server(item); if(ret==ENOENT){ if(ISSET(cnp->cn_flags, ISLASTCN)){ err = cache_lookup(cnp); if(err==CACHE_HIT){ /*item rm'd elsewhere, remove local vnode*/ vn_revoke(vp); cache_purge(dvp); vnode_update_identity(VNODE_UPDATE_PURGE); } else if(err==ENOENT) cache_purge_negatives(dvp); else if(cnp->cn_nameiop==CREATE||cnp->cn_nameiop==RENAME) ret = EJUSTRETURN; else if(ISSET(cnp->cn_flags, MAKEENTRY)) cache_enter(dvp, NULLVP, cnp); } } vn_revoke() ends up calling vnop_reclaim(), which in turn does the following: remove_np_from_hashtable(); vnode_removefsref(vp); vnode_clearfsnode(vp); cache_purge(vp); The above does prevent kernel panics from happening when clicking on files or directories that were removed elsewhere in Finder. On stepping back to the parent directory, the removed item is no longer displayed. However, it's still showing in the terminal when queried with ls(1) or stat(1). I've instrumented relevant parts of code with debugging print statements, but I still don't understand why this is happening. Any pointers on what I'm missing would be greatly appreciated.

Bug report FB16082106 submitted. Thanks.

vnode_if.h distributed with Kernel.framework does not define struct vnop_monitor_args: % grep -c 'struct vnop_monitor_args' `xcrun --show-sdk-path`/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/vnode_if.h 0 Because it's defined as a symbol that is private to kernel. Even if I downloaded vnode_if.h from the kernel sources, I wouldn't be able to link against com.apple.kpi.private, would I? All other vnop_name_args structures that aren't defined as kernel private are available in vnode_if.h distributed with Kernel.framework: % awk '/^struct vnop_.*_args/{print $2}' `xcrun --show-sdk-path`/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/vnode_if.h | sort vnop_access_args vnop_advlock_args vnop_allocate_args vnop_blktooff_args vnop_blockmap_args vnop_bwrite_args vnop_clonefile_args vnop_close_args vnop_copyfile_args vnop_create_args vnop_exchange_args vnop_fsync_args vnop_getattr_args vnop_getattrlistbulk_args vnop_getnamedstream_args vnop_getxattr_args vnop_inactive_args vnop_ioctl_args vnop_kqfilt_add_args vnop_kqfilt_remove_args vnop_link_args vnop_listxattr_args vnop_lookup_args vnop_makenamedstream_args vnop_mkdir_args vnop_mknod_args vnop_mmap_args vnop_mmap_check_args vnop_mnomap_args vnop_offtoblk_args vnop_open_args vnop_pagein_args vnop_pageout_args vnop_pathconf_args vnop_read_args vnop_readdir_args vnop_readdirattr_args vnop_readlink_args vnop_reclaim_args vnop_remove_args vnop_removenamedstream_args vnop_removexattr_args vnop_rename_args vnop_renamex_args vnop_revoke_args vnop_rmdir_args vnop_searchfs_args vnop_select_args vnop_setattr_args vnop_setlabel_args vnop_setxattr_args vnop_strategy_args vnop_symlink_args vnop_verify_args vnop_whiteout_args vnop_write_args Samba client is Apple's own implementation, and therefore private symbols are available to it. Which is not the case with the filesystem I'm implementing. Or were you suggesting that there is a way to access those symbols from 3rd party filesystems?

Thanks for a very detailed response. Regrettably, VNOP_MONITOR is off limits to 3rd party filesystems, as it's a XNU_KERNEL_PRIVATE symbol. I'll have to find another way of dealing with this. Thanks.

I've observed the following behaviour both in Finder and on command line with respect to my initial query. In Finder, I have to switch between windows for vnop_getattr() to be called on the parent directory and the filesystem changes made on a different machine to be picked up. On command line, running % ls -ld /parent/directory results in vnop_getattr() to be called and the changes to be picked up. The same behaviour is observed when, e.g., writing files. The file size doesn't get updated in real time. I have to switch between windows for file size changes to be picked up. What am I missing? Thanks.

My filesystem's vnop_rmdir calls cache_purge() and vnode_recycle() on RPC returning success. The v_label associated with the vnode of the directory being removed is set to NULL and free()'d via the vnode_recycle() call path: vnode_recycle()->vnode_drop_and_unlock()->vnode_drop_internal(): vnode_drop_internal: vp->v_label = NULL; if (tmpl) { mac_vnode_label_free(tmpl); } So, client A, residing on machine-a, is still referencing the vnode that just got recycled by client-B residing on machine-b. Seeing that mac_vnode_check_getattrlist() gets called before vnode_getattr() is, which calls my filesystem's vnop_getattr(), how do I make client A, and any other machine with a mount point to my filesystem, become aware of vnodes getting recycled by other clients on the grid?

When dtrace(1)-ing archive extraction over smbfs I overlooked a vnop_pathconf call, which turned out to be the missing link. 0 -> smbfs_vnop_pathconf AUHelperService-1676 -> extractme a_name:10 ;_PC_NAME_CHARS_MAX 0 <- smbfs_vnop_pathconf AUHelperService-1676 -> extractme a_name:10 -> 255 0 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0 0 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> 2 ;ENOENT 0 -> smbfs_vnop_lookup AUHelperService-1676 -> TheRooT/extractme/misc 2 nameiop:0 0 <- smbfs_vnop_lookup AUHelperService-1676 -> TheRooT/extractme/misc 2 -> 0 0 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0 0 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> 2 0 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0 0 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> 2 0 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc nameiop:0x2 ;DELETE 0 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc -> 0 0 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0x3 ;RENAME 1 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> -2 ;EJUSTRETURN 1 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc nameiop:0x2 1 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc -> 0 1 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0x3 2 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> -2 2 -> smbfs_vnop_rename AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc -> extractme/nil 2 <- smbfs_vnop_rename AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc -> extractme/nil -> 0 1 -> smbfs_vnop_lookup AUHelperService-1676 -> TheRooT/extractme/misc 2 nameiop:0 2 <- smbfs_vnop_lookup AUHelperService-1676 -> TheRooT/extractme/misc 2 -> 0 2 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0 3 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> 0 With vnop_pathconf implemented archive extraction over the same directory multiple times on my custom filesystem works as expected. Thanks.

Thanks for your input. No, my filesystem is not based on FUSE. It's a port of a distributed filesystem from a UNIX-like OS to macOS. The original implementation mostly uses locking to protect the RPCs involved in each file operation. On macOS, this doesn't seem sufficient. Thus my query. The Linux kernel documentation for VFS describes the locking rules that must be followed by all filesystems. FreeBSD filesystems extensively employ locking in their VFS implementations. So I assumed that macOS must have its own locking rules for VFS as well. It would be helpful to understand what they are, so my filesystem can adhere to them properly.

Turns out, it was my lookup algorithm that was to blame. Once adjusted as described here rm(1) was no longer complaining and running to completion. Thanks for your help.

Thanks a lot for the tip Kevin. Basic testing does work fine. terminal1: % echo line1 >file.txt terminal1: % ed file.txt ;open file.txt in ed terminal2: % rm file.txt terminal1: % ed file.txt ;close file.txt in ed dtrace output for terminal tests Finder 1. Create file.rtf using TextEdit. 2. Close file.rtf using TextEdit. 3. Open file.rtf using TextEdit. 4. Remove file.rtf via Finder. 5. Close file.rtf using TextEdit. dtrace output for Finder tests Onto a debugging session of rm(1) as you suggested.

Thanks for replying. I had a closer look at the signature of select and realised that there was no way I knew of that I could get a file descriptor to pass into it within the kernel.