BTW, how much memory does this machine have? Is it 32g or 16g? It's 36Gb. % expr `sysctl -n hw.memsize` / 1024 / 1024 / 1024 36 The bottom line here is that you ultimately need to constrain how quickly you complete write calls to be roughly in line with your underlying ability to transfer data. My attempt at this was through modifying the socket polling algorithm. The decision on whether it's a read or a write event that's occurred is taken in the sock_upcall callback. The socket's receive buffer is queried about whether there's any data available via a call to sock_getsockopt(SO_NREAD). If there is, a thread waiting to read data is sent a read event wakeup() call. Otherwise, the algorithm considers this to be a write event and a thread waiting to write data is sent a wakeup() call. This doesn't take into account whether or not there's room in the send buffer. It's a 'poor' man's socket polling algo, due to functions like sock_setupcalls(), soreadable(), and sowriteable() being a part of the private API. To try and bring the rate the data is being written in line with the rate it's being transferred, I tried modifying the write part of the algorithm by considering it a write event only if there's no more data left in the socket's send buffer via a call to sock_getsockopt(SO_NWRITE). That didn't help remedy the problem. I can't think of other ways of achieving this at the moment.

Does the panic happen when you're just the source and/or destination or only when your "both"? I suspect it will happen when you're just the destination and won't happen when you''re just the source, but I'd like to confirm that. You were right in assuming that the panic occurred when my filesystem was the destination. I was able to verify that. How fast is the I/O path back to the server? Are you saturating that connection? The connection is not likely to be saturated as this is a 100Gb Link on a thunderbolt 5 interface. Is the I/O pipeline here simply "read compressed file from network-> decompress data-> write data out to network"? Or is there any intermediary in that process? The I/O pipeline is as you described it with no intermediary involved. What actually makes it "back" to the server before the panic occurs? How much data were you actually able to write? On two subsequent runs, around 41-42 Gb out of 64Gb of data were written before the panic ensued. du -smx ./25116_CINEGRAPHER_MARCCAIN 41303 ./25116_CINEGRAPHER_MARCCAIN du -smx ./25116_CINEGRAPHER_MARCCAIN 42600 ./25116_CINEGRAPHER_MARCCAIN How does your write process actually "work"? Is there anything that would limit/constrain how much data you have pending (waiting to be sent over the network)? The source uio_t buffer passed into vnop_write() is userspace. Before passing it down to sock_send() which operates on kernel resident memory buffers, we create a kernelspace copy of the userspace uio_t buffer of size equal to uio_resid(uspace_uio) with copying performed by uiomove() incrementally in chunks equal to either the value of the amount of data left in the userspace buffer or the value of the kernel's copysize_limit_panic, whichever happens to be the smaller of the two. The kernelspace uio_t buffer is further split up into smaller chunks of data pertinent to the filesystem design which end up being passed into sock_send(). Reading is done in a similar fashion, the only difference being the use of sock_receive_mbuf() in stead of sock_receive() which uses an uio_t buffer instead of an mbuf. I'm onto the debugging strategies you suggested now. I'll report back on my findings as they emerge. Thanks once again for all your help. Hopefully, we'll be able to resolve this soon.

Thanks very much for looking into this. Here are the points you wanted clarified. How much space is available on the boot volume and is there ANY possibility That it's becoming constrained? That doesn't seem to be the case. See Boot volume size info what's the content you're actually decompressing? One (or several) large files or a bunch of small ones? It is a bunch of files of varying sizes. Here's a List of zip file contents Is your file system the source or the target for the zip file? It's actually both. what's actually providing the storage for your file system? Is it possible that some supporting component has failed? It is a network filesystem with I/O happening via RPC calls. Internally, the filesystem implements a socket polling algorithm that relies on the use of sock_upcall callback that's passed into sock_socket() on socket creation. The backing store on the remote server has been verified to have ample disk space available. Do let me know if there's anything else I can provide that would help identify the cause of the problem.

Hello Kevin, I was wondering if you've had a chance to take a look at FB17497530 yet? Thanks.

Thanks a lot for your really helpful and detailed responses. Two-machine debugging on Apple Silicon has certain limitations, which are worded in the KDK Readme as follows: Note: Apple silicon doesn’t support active kernel debugging. You may inspect the current state of the kernel when it is halted due to a panic or NMI. However, you cannot set breakpoints, continue code execution, step into code, step over code, or step out of the current instruction. x86-64 is better suited for remote kernel debugging. This particular panic doesn't occur on x86-64 though. Only on Apple Silicon. I could try and set up a core dump server/client and see how that pans out. Before I go ahead and file a bug report you suggested by way of submitting a full panic log, you mentioned about ES. In the full log, I found a reference to this: "name":"AFKDCPEXTEndpoint9"},"1597":{"id":1597,"system_usec":104537416,"schedPriority":91,"state":["TH_RUN"],"snapshotFlags":["kKernel64_p","kThreadIOPassive","kThreadOnCore"],"user_usec":0,"kernelFrames":[[0,377776],[0,9311856],[10,23512],[10,18668],[10,6612],[11,6952],[0,1810716],[0,31032],[0,1395328],[0,1395328],[0,6300680],[0,6673156],[0,1049164],[0,1389844],[0,68612]],"basePriority":91,"lastMadeRunnableTime":3.1102084579999998,"userTime":0,"lastRunTime":3.1102656249999998,"systemTime":104.537416333, Which corresponds to the panicked thread id: Panicked thread: 0xfffffe31ffe31fc0, backtrace: 0xfffffe81838478d0, tid: 1597 That thread name, AFKDCPEXTEndpoint9, wouldn't a part of the ES API you were referring to, would it?

Thanks. That makes perfect sense now. I got the same symbolicated output on my system following your instructions. [quote='836958022, DTS Engineer, /thread/782023?answerId=836958022#836958022'] Are they from your KEXT and you just didn't include the KEXT address info? [/quote] Here are the addresses given in the panic log: Kernel Extensions in backtrace That particular kernel panic is reproducible whenever a very large zip file is being extracted over my filesystem. With watchdog panics, is there a way of telling what caused watchdogd to emit a panic?

Found the answer I was looking for by dtrace(1)'ing SMBClient. It calls ubc_msync(vp, 0, ubc_getsize(vp), NULL, UBC_INVALIDATE); in smbfs_vnop_open_common(). The following commentary explains the reasoning behind this.

Thanks. Just a couple of points to clarify. atos -arch arm64e -o <symbol file path) -l <load address + (0x4000/vmaddr)> <address to symbol> In that particular panic log, there's not load address specified for my kext. What address do I specify in that case? 0x4000/vmaddr is that divided by vmaddr or is that saying that 0x4000 is the value of vmaddr?

Thanks for your response. I've downloaded and extracted Kernel_Debug_Kit_15.4_build_24E248.dmg. Here's what I've tried so far to sybolicate the sample kernel panic log. Note: the kernel passed in to the -o flag of atos(1) is the one that is found in kernel.release.t8112.dSYM/Contents/Resources/DWARF/kernel.release.t8112. The load address computed as the kernel base addresses given in the kernel panic log minus the corresponding slide atos -o kernel.release.t8112 -arch arm64e -l FFFFFE0007004000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 No output generated The value of Kernel slide given in the kernel panic log passed to the -s flag atos -o kernel.release.t8112 -arch arm64e -s 0000000014610000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 No output generated The value of KernelCache slide given in the kernel panic log passed to the -s flag atos -o kernel.release.t8112 -arch arm64e -s 0000000014608000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 No output generated The value of Kernel text exec slide given in the kernel panic log passed to the -s flag atos -o kernel.release.t8112 -arch arm64e -s 0000000015D88000 0xfffffe001cde8ab8 0xfffffe001cf44894 0xfffffe001cf42a90 0xfffffe001cd938c0 0xfffffe001cde83b0 0xfffffe001d66d670 0xfffffe001d86a708 0xfffffe001d86941c 0xfffffe001d866504 0xfffffe001e234bc8 0xfffffe001cf4611c 0xfffffe001cd93938 0xfffffe001cee0a80 0xfffffe001cee0a80 0xfffffe001d38e408 0xfffffe001d3e9304 0xfffffe001ce8c24c 0xfffffe001cedf514 0xfffffe001cd9cc04 Output: Looks bogus __doprnt.digits (in kernel.release.t8112) + 103398 bootstrap_pagetables (in kernel.release.t8112) + 329876 bootstrap_pagetables (in kernel.release.t8112) + 322192 table_10bits (in kernel.release.t8112) + 10400 __doprnt.digits (in kernel.release.t8112) + 101598 recvmsg_x (in kernel.release.t8112) + 332 IOMachPort::noMoreSendersForObject(OSObject*, unsigned int, unsigned int*) (in kernel.release.t8112) + 440 IOUserClient::init(OSDictionary*) (in kernel.release.t8112) + 0 IOServiceMessageUserNotification::MetaClass::~MetaClass() (in kernel.release.t8112) + 528 bootstrap_pagetables (in kernel.release.t8112) + 336156 table_10bits (in kernel.release.t8112) + 10520 ropagetable_end (in kernel.release.t8112) + 2688 ropagetable_end (in kernel.release.t8112) + 2688 ipsec_cleanup_family (in kernel.release.t8112) + 920 pf_test (in kernel.release.t8112) + 3700 ropagetable_begin (in kernel.release.t8112) + 324884 nfU16InvMasks (in kernel.release.t8112) + 112 Can you please instruct me how to symbolicate my sample kernel panic log properly.

That sounds reasonable. Thanks a lot for your really valuable input.

Thanks for all your help.

Thanks very much for your response. I did manage to get it to work by purging the local name cache as well as the hash table where FSNodes are being kept in vnop_getattr() on receiving a notification from the server of a directory entry no longer existing. I do appreciate that it's not an ideal solution though for the reasons you described. Apple have made VNOP_MONITOR() available as a public symbol in macOS 15.4 beta in response to the bug report I submitted, FB16082106. Is there anywhere I could look for VNOP_MONITOR()+vnode_notify() usage examples other than the SMBClient and Darwin/XNU sources as it's not immediately clear how remote filesystem changes are propagated across all connected clients through the use of those functions?

With VNOP_MONITOR() still unavailable I thought I'd explore other alternatives. Passing VFS_TBLNOMACLABEL into vfs_fsadd() has revealed that the issue was with the vnode being NULL when accessing directories that were removed on other client machines. Further debugging has confirmed that the vnode was being set to NULL via a call to cache_enter() with cn_flags set to MAKEENTRY in vnop_lookup() on receiving a response from the server saying that the directory was no longer there. Dtrace(1)'ing vnode's mac labels showed that for my filesystem they were set to NULL at all times whether or not VFS_TBLNOMACLABEL was set. My filesystem being distributed, I have the benefit of querying the server about the presence of files or directories in my vnop_lookup(). What I attempted next is modify my lookup algorithm to remove the local vnode that represents a file or directory that the server reports as no longer existing, i.e. removed by another client machine. Below is a pseudocode for the part of my lookup algorithm that deals with negative server responses. ret = query_server(item); if(ret==ENOENT){ if(ISSET(cnp->cn_flags, ISLASTCN)){ err = cache_lookup(cnp); if(err==CACHE_HIT){ /*item rm'd elsewhere, remove local vnode*/ vn_revoke(vp); cache_purge(dvp); vnode_update_identity(VNODE_UPDATE_PURGE); } else if(err==ENOENT) cache_purge_negatives(dvp); else if(cnp->cn_nameiop==CREATE||cnp->cn_nameiop==RENAME) ret = EJUSTRETURN; else if(ISSET(cnp->cn_flags, MAKEENTRY)) cache_enter(dvp, NULLVP, cnp); } } vn_revoke() ends up calling vnop_reclaim(), which in turn does the following: remove_np_from_hashtable(); vnode_removefsref(vp); vnode_clearfsnode(vp); cache_purge(vp); The above does prevent kernel panics from happening when clicking on files or directories that were removed elsewhere in Finder. On stepping back to the parent directory, the removed item is no longer displayed. However, it's still showing in the terminal when queried with ls(1) or stat(1). I've instrumented relevant parts of code with debugging print statements, but I still don't understand why this is happening. Any pointers on what I'm missing would be greatly appreciated.

Bug report FB16082106 submitted. Thanks.

vnode_if.h distributed with Kernel.framework does not define struct vnop_monitor_args: % grep -c 'struct vnop_monitor_args' `xcrun --show-sdk-path`/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/vnode_if.h 0 Because it's defined as a symbol that is private to kernel. Even if I downloaded vnode_if.h from the kernel sources, I wouldn't be able to link against com.apple.kpi.private, would I? All other vnop_name_args structures that aren't defined as kernel private are available in vnode_if.h distributed with Kernel.framework: % awk '/^struct vnop_.*_args/{print $2}' `xcrun --show-sdk-path`/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/vnode_if.h | sort vnop_access_args vnop_advlock_args vnop_allocate_args vnop_blktooff_args vnop_blockmap_args vnop_bwrite_args vnop_clonefile_args vnop_close_args vnop_copyfile_args vnop_create_args vnop_exchange_args vnop_fsync_args vnop_getattr_args vnop_getattrlistbulk_args vnop_getnamedstream_args vnop_getxattr_args vnop_inactive_args vnop_ioctl_args vnop_kqfilt_add_args vnop_kqfilt_remove_args vnop_link_args vnop_listxattr_args vnop_lookup_args vnop_makenamedstream_args vnop_mkdir_args vnop_mknod_args vnop_mmap_args vnop_mmap_check_args vnop_mnomap_args vnop_offtoblk_args vnop_open_args vnop_pagein_args vnop_pageout_args vnop_pathconf_args vnop_read_args vnop_readdir_args vnop_readdirattr_args vnop_readlink_args vnop_reclaim_args vnop_remove_args vnop_removenamedstream_args vnop_removexattr_args vnop_rename_args vnop_renamex_args vnop_revoke_args vnop_rmdir_args vnop_searchfs_args vnop_select_args vnop_setattr_args vnop_setlabel_args vnop_setxattr_args vnop_strategy_args vnop_symlink_args vnop_verify_args vnop_whiteout_args vnop_write_args Samba client is Apple's own implementation, and therefore private symbols are available to it. Which is not the case with the filesystem I'm implementing. Or were you suggesting that there is a way to access those symbols from 3rd party filesystems?