Thanks a lot for your really helpful and detailed responses. Two-machine debugging on Apple Silicon has certain limitations, which are worded in the KDK Readme as follows: Note: Apple silicon doesn’t support active kernel debugging. You may inspect the current state of the kernel when it is halted due to a panic or NMI. However, you cannot set breakpoints, continue code execution, step into code, step over code, or step out of the current instruction. x86-64 is better suited for remote kernel debugging. This particular panic doesn't occur on x86-64 though. Only on Apple Silicon. I could try and set up a core dump server/client and see how that pans out. Before I go ahead and file a bug report you suggested by way of submitting a full panic log, you mentioned about ES. In the full log, I found a reference to this: "name":"AFKDCPEXTEndpoint9"},"1597":{"id":1597,"system_usec":104537416,"schedPriority":91,"state":["TH_RUN"],"snapshotFlags":["kKernel64_p","kThreadIOPassive","kThreadOnCore"],"user_usec":0,"kernelFrames":[[0,377776],[0,9311856],[10,23512],[10,18668],[10,6612],[11,6952],[0,1810716],[0,31032],[0,1395328],[0,1395328],[0,6300680],[0,6673156],[0,1049164],[0,1389844],[0,68612]],"basePriority":91,"lastMadeRunnableTime":3.1102084579999998,"userTime":0,"lastRunTime":3.1102656249999998,"systemTime":104.537416333, Which corresponds to the panicked thread id: Panicked thread: 0xfffffe31ffe31fc0, backtrace: 0xfffffe81838478d0, tid: 1597 That thread name, AFKDCPEXTEndpoint9, wouldn't a part of the ES API you were referring to, would it?

Thanks. That makes perfect sense now. I got the same symbolicated output on my system following your instructions. [quote='836958022, DTS Engineer, /thread/782023?answerId=836958022#836958022'] Are they from your KEXT and you just didn't include the KEXT address info? [/quote] Here are the addresses given in the panic log: Kernel Extensions in backtrace That particular kernel panic is reproducible whenever a very large zip file is being extracted over my filesystem. With watchdog panics, is there a way of telling what caused watchdogd to emit a panic?

Found the answer I was looking for by dtrace(1)'ing SMBClient. It calls ubc_msync(vp, 0, ubc_getsize(vp), NULL, UBC_INVALIDATE); in smbfs_vnop_open_common(). The following commentary explains the reasoning behind this.

Thanks. Just a couple of points to clarify. atos -arch arm64e -o <symbol file path) -l <load address + (0x4000/vmaddr)> <address to symbol> In that particular panic log, there's not load address specified for my kext. What address do I specify in that case? 0x4000/vmaddr is that divided by vmaddr or is that saying that 0x4000 is the value of vmaddr?

Thanks for your response. I've downloaded and extracted Kernel_Debug_Kit_15.4_build_24E248.dmg. Here's what I've tried so far to sybolicate the sample kernel panic log. Note: the kernel passed in to the -o flag of atos(1) is the one that is found in kernel.release.t8112.dSYM/Contents/Resources/DWARF/kernel.release.t8112. The load address computed as the kernel base addresses given in the kernel panic log minus the corresponding slide atos -o kernel.release.t8112 -arch arm64e -l FFFFFE0007004000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 No output generated The value of Kernel slide given in the kernel panic log passed to the -s flag atos -o kernel.release.t8112 -arch arm64e -s 0000000014610000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 No output generated The value of KernelCache slide given in the kernel panic log passed to the -s flag atos -o kernel.release.t8112 -arch arm64e -s 0000000014608000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 No output generated The value of Kernel text exec slide given in the kernel panic log passed to the -s flag atos -o kernel.release.t8112 -arch arm64e -s 0000000015D88000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 Output: Looks bogus __doprnt.digits (in kernel.release.t8112) + 103398 bootstrap_pagetables (in kernel.release.t8112) + 329876 bootstrap_pagetables (in kernel.release.t8112) + 322192 table_10bits (in kernel.release.t8112) + 10400 __doprnt.digits (in kernel.release.t8112) + 101598 recvmsg_x (in kernel.release.t8112) + 332 IOMachPort::noMoreSendersForObject(OSObject*, unsigned int, unsigned int*) (in kernel.release.t8112) + 440 IOUserClient::init(OSDictionary*) (in kernel.release.t8112) + 0 IOServiceMessageUserNotification::MetaClass::~MetaClass() (in kernel.release.t8112) + 528 bootstrap_pagetables (in kernel.release.t8112) + 336156 table_10bits (in kernel.release.t8112) + 10520 ropagetable_end (in kernel.release.t8112) + 2688 ropagetable_end (in kernel.release.t8112) + 2688 ipsec_cleanup_family (in kernel.release.t8112) + 920 pf_test (in kernel.release.t8112) + 3700 ropagetable_begin (in kernel.release.t8112) + 324884 nfU16InvMasks (in kernel.release.t8112) + 112 Can you please instruct me how to symbolicate my sample kernel panic log properly.

That sounds reasonable. Thanks a lot for your really valuable input.

Thanks for all your help.

Thanks very much for your response. I did manage to get it to work by purging the local name cache as well as the hash table where FSNodes are being kept in vnop_getattr() on receiving a notification from the server of a directory entry no longer existing. I do appreciate that it's not an ideal solution though for the reasons you described. Apple have made VNOP_MONITOR() available as a public symbol in macOS 15.4 beta in response to the bug report I submitted, FB16082106. Is there anywhere I could look for VNOP_MONITOR()+vnode_notify() usage examples other than the SMBClient and Darwin/XNU sources as it's not immediately clear how remote filesystem changes are propagated across all connected clients through the use of those functions?

With VNOP_MONITOR() still unavailable I thought I'd explore other alternatives. Passing VFS_TBLNOMACLABEL into vfs_fsadd() has revealed that the issue was with the vnode being NULL when accessing directories that were removed on other client machines. Further debugging has confirmed that the vnode was being set to NULL via a call to cache_enter() with cn_flags set to MAKEENTRY in vnop_lookup() on receiving a response from the server saying that the directory was no longer there. Dtrace(1)'ing vnode's mac labels showed that for my filesystem they were set to NULL at all times whether or not VFS_TBLNOMACLABEL was set. My filesystem being distributed, I have the benefit of querying the server about the presence of files or directories in my vnop_lookup(). What I attempted next is modify my lookup algorithm to remove the local vnode that represents a file or directory that the server reports as no longer existing, i.e. removed by another client machine. Below is a pseudocode for the part of my lookup algorithm that deals with negative server responses. ret = query_server(item); if(ret==ENOENT){ if(ISSET(cnp->cn_flags, ISLASTCN)){ err = cache_lookup(cnp); if(err==CACHE_HIT){ /*item rm'd elsewhere, remove local vnode*/ vn_revoke(vp); cache_purge(dvp); vnode_update_identity(VNODE_UPDATE_PURGE); } else if(err==ENOENT) cache_purge_negatives(dvp); else if(cnp->cn_nameiop==CREATE||cnp->cn_nameiop==RENAME) ret = EJUSTRETURN; else if(ISSET(cnp->cn_flags, MAKEENTRY)) cache_enter(dvp, NULLVP, cnp); } } vn_revoke() ends up calling vnop_reclaim(), which in turn does the following: remove_np_from_hashtable(); vnode_removefsref(vp); vnode_clearfsnode(vp); cache_purge(vp); The above does prevent kernel panics from happening when clicking on files or directories that were removed elsewhere in Finder. On stepping back to the parent directory, the removed item is no longer displayed. However, it's still showing in the terminal when queried with ls(1) or stat(1). I've instrumented relevant parts of code with debugging print statements, but I still don't understand why this is happening. Any pointers on what I'm missing would be greatly appreciated.

Bug report FB16082106 submitted. Thanks.

vnode_if.h distributed with Kernel.framework does not define struct vnop_monitor_args: % grep -c 'struct vnop_monitor_args' `xcrun --show-sdk-path`/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/vnode_if.h 0 Because it's defined as a symbol that is private to kernel. Even if I downloaded vnode_if.h from the kernel sources, I wouldn't be able to link against com.apple.kpi.private, would I? All other vnop_name_args structures that aren't defined as kernel private are available in vnode_if.h distributed with Kernel.framework: % awk '/^struct vnop_.*_args/{print $2}' `xcrun --show-sdk-path`/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/vnode_if.h | sort vnop_access_args vnop_advlock_args vnop_allocate_args vnop_blktooff_args vnop_blockmap_args vnop_bwrite_args vnop_clonefile_args vnop_close_args vnop_copyfile_args vnop_create_args vnop_exchange_args vnop_fsync_args vnop_getattr_args vnop_getattrlistbulk_args vnop_getnamedstream_args vnop_getxattr_args vnop_inactive_args vnop_ioctl_args vnop_kqfilt_add_args vnop_kqfilt_remove_args vnop_link_args vnop_listxattr_args vnop_lookup_args vnop_makenamedstream_args vnop_mkdir_args vnop_mknod_args vnop_mmap_args vnop_mmap_check_args vnop_mnomap_args vnop_offtoblk_args vnop_open_args vnop_pagein_args vnop_pageout_args vnop_pathconf_args vnop_read_args vnop_readdir_args vnop_readdirattr_args vnop_readlink_args vnop_reclaim_args vnop_remove_args vnop_removenamedstream_args vnop_removexattr_args vnop_rename_args vnop_renamex_args vnop_revoke_args vnop_rmdir_args vnop_searchfs_args vnop_select_args vnop_setattr_args vnop_setlabel_args vnop_setxattr_args vnop_strategy_args vnop_symlink_args vnop_verify_args vnop_whiteout_args vnop_write_args Samba client is Apple's own implementation, and therefore private symbols are available to it. Which is not the case with the filesystem I'm implementing. Or were you suggesting that there is a way to access those symbols from 3rd party filesystems?

Thanks for a very detailed response. Regrettably, VNOP_MONITOR is off limits to 3rd party filesystems, as it's a XNU_KERNEL_PRIVATE symbol. I'll have to find another way of dealing with this. Thanks.

I've observed the following behaviour both in Finder and on command line with respect to my initial query. In Finder, I have to switch between windows for vnop_getattr() to be called on the parent directory and the filesystem changes made on a different machine to be picked up. On command line, running % ls -ld /parent/directory results in vnop_getattr() to be called and the changes to be picked up. The same behaviour is observed when, e.g., writing files. The file size doesn't get updated in real time. I have to switch between windows for file size changes to be picked up. What am I missing? Thanks.

My filesystem's vnop_rmdir calls cache_purge() and vnode_recycle() on RPC returning success. The v_label associated with the vnode of the directory being removed is set to NULL and free()'d via the vnode_recycle() call path: vnode_recycle()->vnode_drop_and_unlock()->vnode_drop_internal(): vnode_drop_internal: vp->v_label = NULL; if (tmpl) { mac_vnode_label_free(tmpl); } So, client A, residing on machine-a, is still referencing the vnode that just got recycled by client-B residing on machine-b. Seeing that mac_vnode_check_getattrlist() gets called before vnode_getattr() is, which calls my filesystem's vnop_getattr(), how do I make client A, and any other machine with a mount point to my filesystem, become aware of vnodes getting recycled by other clients on the grid?

When dtrace(1)-ing archive extraction over smbfs I overlooked a vnop_pathconf call, which turned out to be the missing link. 0 -> smbfs_vnop_pathconf AUHelperService-1676 -> extractme a_name:10 ;_PC_NAME_CHARS_MAX 0 <- smbfs_vnop_pathconf AUHelperService-1676 -> extractme a_name:10 -> 255 0 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0 0 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> 2 ;ENOENT 0 -> smbfs_vnop_lookup AUHelperService-1676 -> TheRooT/extractme/misc 2 nameiop:0 0 <- smbfs_vnop_lookup AUHelperService-1676 -> TheRooT/extractme/misc 2 -> 0 0 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0 0 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> 2 0 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0 0 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> 2 0 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc nameiop:0x2 ;DELETE 0 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc -> 0 0 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0x3 ;RENAME 1 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> -2 ;EJUSTRETURN 1 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc nameiop:0x2 1 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc -> 0 1 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0x3 2 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> -2 2 -> smbfs_vnop_rename AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc -> extractme/nil 2 <- smbfs_vnop_rename AUHelperService-1676 -> extractme.sb-27bcb433-Yj4plp/misc -> extractme/nil -> 0 1 -> smbfs_vnop_lookup AUHelperService-1676 -> TheRooT/extractme/misc 2 nameiop:0 2 <- smbfs_vnop_lookup AUHelperService-1676 -> TheRooT/extractme/misc 2 -> 0 2 -> smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 nameiop:0 3 <- smbfs_vnop_lookup AUHelperService-1676 -> extractme/misc 2 -> 0 With vnop_pathconf implemented archive extraction over the same directory multiple times on my custom filesystem works as expected. Thanks.