Installing ZFSFSKit.appex ? /Library/ExtensionKit/Extensions/ Substituting real Mach-O (libtool wrapper ? .libs/ZFSFSKit) Installing zfs.fs ? /Library/Filesystems/ mount_zfs: Mach-O 64-bit executable arm64 Done. Signing (before pluginkit, so it sees a valid signature)... Re-signing /Library/ExtensionKit/Extensions/ZFSFSKit.appex ad-hoc (no identity). Note: requires amfi_get_out_of_my_way=1 in boot-args. Team ID: ADHOC /Library/ExtensionKit/Extensions/ZFSFSKit.appex: replacing existing signature Done. Signature: Identifier=org.openzfsonosx.filesystems.zfs.fsext Signature=adhoc TeamIdentifier=not set Entitlements: <?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.application-identifier</key><string>ADHOC.org.openzfsonosx.filesystems.zfs.fsext</string><key>com.apple.developer.fskit.fsmodule</key><true/><key>com.apple.developer.team-identifier</key><string>ADHOC</string><key>com.apple.security.app-sandbox</key><true/></dict></plist> Registering with pluginkit... pluginkit -a done. Restarting fskitd... # sudo pluginkit -v -m -p com.apple.fskit.fsmodule + org.openzfsonosx.filesystems.zfs.fsext((null)) 6A12A41280FB-4190-B957-FA94DC89BB1E 2026-05-29 01:17:58 +0000 /Library/ExtensionKit/Extensions/ZFSFSKit.appex # sudo mkdir /Volumes/tank # sudo mount -F -t zfs /dev/disk4 /Volumes/tank # ls -la /Volumes/tank total 3 drwxr-xr-x 3 lundman staff 4 May 29 09:21 . drwxr-xr-x 4 root wheel 128 May 29 10:18 .. -rw-r--r-- 1 lundman staff 11 May 29 09:21 file.txt drwxr-xr-x 2 lundman staff 2 May 29 09:21 HelloWorld # cat /Volumes/tank/file.txt HelloWorld Even though FSKit isn't quite ready, I built a proof-of-concept FSKit extension to understand what the migration path looks like. This post shares what we got working, specific technical findings that weren't documented, and the gaps we hit that would need Apple's attention for a production implementation. Luckily, OpenZFS already compiles in userland for the "zdb" utility so not much work was required on that side. There were certain amount of desperation applied when we came across hurdles, so possibly some assumptions we formed are not correct. (We didn't go back and confirm the problem after it started working).

When I have to pull in hundreds of commits from upstream, I like to try to make sure things still compile - frequently, to try to limit how far I need to go to back fix things. One issue is missing symbols in the kext, since you won't know until you try to load the kext. And loading the kext each commit is not realistic. So I went and made up a call to something that does not exist, in my case, strqcmp(). I could not get the various tools like kmutil libraries --all-symbols to print out that this function was going to fail, so I wrote a little script (thanks ChatGPT); ./scripts/kpi_check.py --arch arm64e -k module/os/macos/zfs.kext/ First missing symbols: _crfree _crhold _strqcmp Hurrah. But sadly, my brain was then curious as to why crhold() and crfree() work. Worked for years. Only dtrace calls them in XNU sources but otherwise not mentioned there, not listed in my frameworks, nm is not seeing it. Somewhat of a rabbit hole. I don't even need to know, it does work after all. I should just let it go right? and yet... how does it work? My best guess is a symbols.alias pointing it to kauth_cred_ref() somewhere? Maybe? Anyway, pretty low priority but it's an itch...

So if one were to start the attempt of porting an existing kext VFS filesystem, to use the new FSKit (Since presumably kexts could go away), how would that look now? Is it ready? Are there any samples out there that already works (Filesystem using FSKit) ? How is the documentation? ChatGPT did not seem to know much at all. What would be Apple's reception to that? How flexible is FSKit ? Is it locked to the idea of a mount is connected to a physical device (or partition)? Or is it more virtual, in that I will have a pool of disks, and present 1, or many, mount points?

So upstream went and added a mutex_enter_interruptible() which Linux calls mutex_lock_interruptible() and FreeBSD sx_xlock_sig(lock). I was simply going to point it to lck_mtx_lock() and call it a day and ignoring the interruptible bit, but I am curious if there is a way to achieve something similar on XNU. In this case, to be able to hit ^C in userland, get a signal, and have lck_mtx_lock() or variant, giveup and return error.

Trying to get some minimum development working again, I've been waiting to be able to macOS in VMs on M1. Currently both VirtualBuddy, and UTM, can install macOS, I can go to Recovery Boot to disable SIP and enable 3rd party extensions. My M1 runs: ProductVersion: 13.0 BuildVersion: 22A5331f I've tested VM macOS versions of Monterey and Ventura. Here is my old kext (known to be working) loaded on M1 (Ventura) bare-metal 250 0 0xfffffe0006b70000 0x862ac 0x862ac org.openzfsonosx.zfs (2.1.0) BE4DF1D3-FF77-3E58-BC9A-C0B8E175DD97 &lt;21 7 5 4 3 1&gt; The same pkg, using the same steps in the VM, will after clicking Allow, ask to reboot (suspiciously fast), then come up with: System Extension Error: An error occurred with your system extensions during startup and they need to be rebuilt before they can be used. Of course clicking Allow just does the same, reboot, fail, ask to approve again, reboot..fail... Directly on the hardware, the dialog "rebuilding cache" pops up for a few seconds, but with the VMs I do not see it. I'm unfamiliar with the new system, so I'm not sure which log files to look at, but here is the output from kmtuil log, both at Allow and after reboot: https://www.lundman.net/kmutil-log.txt If I was going to make an uneducated guess and pull out some lines by random, maybe: 2022-08-29 20:01:13.169897+0900 0x251 Error 0x0 100 0 kernelmanagerd: Kcgen roundtrip failed with: Boot policy error: Error creating linked manifest: code BOOTPOLICY_ERROR_ACM 2022-08-29 20:01:13.170200+0900 0x251 Error 0x0 100 0 kernelmanagerd: Kcgen roundtrip failed checkpoint saveAuxkc: status:error fatalError:Optional("Boot policy error: Error creating linked manifest: code BOOTPOLICY_ERROR_ACM") 2022-08-29 20:01:13.170201+0900 0x251 Error 0x0 100 0 kernelmanagerd: Kcgen roundtrip failed: missing last checkpoint or errors found 2022-08-29 20:01:13.170242+0900 0x251 Default 0x0 100 0 kernelmanagerd: Deleting Preboot content Any work arounds? Loading kexts on my only M1 is a hard way to develop.

So, M2 came with 22D68 on it, and closest KDK is 22D49. In the Intel days, I would just copy the KDK 22D49 to the system and reboot into it. Can we still achieve this on arm? Recovery boot, disable authenticated-root, something-something snapshot.. ? Anyone figured it out already?

Since it is Private exported, can't use it directly.

So we have produced kexts that run well, on Intel and Arm64, on (for example) an MBP/M1 (all macOS currently available), and MacStudio (Monterey). But on Monterey + Ventura it enters a boot panic loop. One example is: "build" : "macOS 13.1 (22C5033e)", "product" : "Mac13,2", "socId" : "0x00006002", "kernel" : "Darwin Kernel Version 22.2.0: Sun Oct 16 18:09:52 PDT 2022; root:xnu-8792.60.32.0.1~11\/RELEASE_ARM64_T6000", "incident" : "8D3814E3-DCBB-42A6-AACF-C37F66D6BBC8", "crashReporterKey" : "FF922DC9-99E1-68B9-75FB-9427F2BBF431", "date" : "2022-10-28 00:12:53.22 +0100", "panicString" : "panic(cpu 6 caller 0xfffffe001e4b11e8): \"apciec[pcic2-bridge]::handleInterrupt: Request address is greater than 32 bits linksts=0x99000001 pcielint=0x02220060 linkcdmsts=0x00000000 (ltssm 0x11=L0)\\n\" @AppleT8103PCIeCPort.cpp:1301\n Debugger message: panic\nMemory ID: 0x6\nOS release type: User\nOS version: 22C5033e\nKernel version: Darwin Kernel Version 22.2.0: Sun Oct 16 18:09:52 PDT 2022; root:xnu-8792.60.32.0.1~11\/RELEASE_ARM64_T6000\nFileset Kernelcache UUID: D767CC1C43ABBCC48AD47B6010804F47\nKernel UUID: 99C80004-214F-342C-ADF2-402BC1EAC155\nBoot session UUID: 8D3814E3-DCBB-42A6-AACF-C37F66D6BBC8\niBoot version: iBoot-8419.60.31\nsecure boot?: YES\nroots installed: 0\nPaniclog version: 14\nKernelCache slide: 0x00000000149f4000\nKernelCache base: 0xfffffe001b9f8000\nKernel slide: 0x0000000015c54000\nKernel text base: 0xfffffe001cc58000\nKernel text exec slide: 0x0000000015d40000\nKernel text exec base: 0xfffffe001cd44000\nmach_absolute_time: 0x1506187a\nEpoch Time: sec usec\n Boot : 0x635b102d 0x0009de48\n Sleep : 0x00000000 0x00000000\n Wake : 0x00000000 0x00000000\n Calendar: 0x635b1035 0x000bfb29\n\nZone info:\n Zone map: 0xfffffe10219f0000 - 0xfffffe30219f0000\n . VM : 0xfffffe10219f0000 - 0xfffffe14ee6bc000\n . RO : 0xfffffe14ee6bc000 - 0xfffffe1688054000\n . GEN0 : 0xfffffe1688054000 - 0xfffffe1b54d20000\n . GEN1 : 0xfffffe1b54d20000 - 0xfffffe20219ec000\n . GEN2 : 0xfffffe20219ec000 - 0xfffffe24ee6b8000\n . GEN3 : 0xfffffe24ee6b8000 - 0xfffffe29bb384000\n . DATA : 0xfffffe29bb384000 - 0xfffffe30219f0000\n Metadata: 0xfffffe3021a00000 - 0xfffffe3029a00000\n Bitmaps : 0xfffffe3029a00000 - 0xfffffe3049a00000\n\nTPIDRx_ELy = {1: 0xfffffe29bae26818 0: 0x0000000000000006 0ro: 0x0000000000000000 }\nCORE 0 PVH locks held: None\nCORE 1 PVH locks held: None\nCORE 2 PVH locks held: None\nCORE 0: PC=0x00000001af0305ec, LR=0x00000001af0304e8, FP=0x000000016dda6200\nCORE 1: PC=0x00000001aeee1c94, LR=0x00000001bb020624, FP=0x000000016e206140\nCORE 2: PC=0xfffffe001d4a15e0, LR=0xfffffe001d4a14cc, FP=0xfffffe8ff2263d30\nCORE 3: PC=0xfffffe001cd9c9f8, LR=0xfffffe001d6eac6c, FP=0xfffffe80212ab920\nCORE 4: PC=0xfffffe001d416704, LR=0xfffffe001d4a265c, FP=0xfffffe802138fd50\nCORE 5: PC=0xfffffe001cd9c974, LR=0xfffffe001f14aca0, FP=0xfffffe8020a4ba50\nCORE 6 is the one that panicked. Check the full backtrace for details.\nCORE 7: PC=0xfffffe001cdbbad0, LR=0xfffffe001cdbbb88, FP=0xfffffe8ff1f6fd90\nCORE 8: PC=0xfffffe001cddfca4, LR=0xfffffe001cddfca4, FP=0xfffffe8020edff00\nCORE 9: PC=0xfffffe001cddfca4, LR=0xfffffe001cddfca4, FP=0xfffffe8ff2197f00\nCORE 10: PC=0xfffffe001cfd1054, LR=0xfffffe001f9d7d14, FP=0xfffffe8ff1f87610\nCORE 11: PC=0x00000001b2064b88, LR=0x00000001b2064af0, FP=0x000000016f3e5860\nCORE 12: PC=0xfffffe001cddfca4, LR=0xfffffe001cddfca4, FP=0xfffffe8021543f00\nCORE 13: PC=0xfffffe001cddfca4, LR=0xfffffe001cddfca4, FP=0xfffffe8020963f00\nCORE 14: PC=0xfffffe001cddfca4, LR=0xfffffe001cddfca4, FP=0xfffffe8020adff00\nCORE 15: PC=0xfffffe001cddfca4, LR=0xfffffe001cddfca4, FP=0xfffffe8020a03f00\nCORE 16: PC=0xfffffe001cddfca4, LR=0xfffffe001cddfca4, FP=0xfffffe8ff1f4bf00\nCORE 17: PC=0xfffffe001cddfca4, LR=0xfffffe001cddfca4, FP=0xfffffe802146bf00\nCORE 18: PC=0xfffffe001cddfca4, LR=0xfffffe001cddfca4, FP=0xfffffe8ff21c7f00\nCORE 19: PC=0xfffffe001cddfca4, LR=0xfffffe001cddfca4, FP=0xfffffe88090fff00\nCompressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space\nPanicked task 0xfffffe168805d678: 0 pages, 1020 threads: pid 0: kernel_task\nPanicked thread: 0xfffffe29bae26818, backtrace: 0xfffffe80213176a0, tid: 285\n\t\t lr: 0xfffffe001cda2adc fp: 0xfffffe8021317710\n\t\t lr: 0xfffffe001cda2884 fp: 0xfffffe8021317790\n\t\t lr: 0xfffffe001cf06a18 fp: 0xfffffe80213177b0\n\t\t lr: 0xfffffe001cef7f88 fp: 0xfffffe8021317820\n\t\t lr: 0xfffffe001cef588c fp: 0xfffffe80213178e0\n\t\t lr: 0xfffffe001cd4b7f8 fp: 0xfffffe80213178f0\n\t\t lr: 0xfffffe001cda220c fp: 0xfffffe8021317ca0\n\t\t lr: 0xfffffe001d5e6b74 fp: 0xfffffe8021317cc0\n\t\t lr: 0xfffffe001e4b11e8 fp: 0xfffffe8021317d80\n\t\t lr: 0xfffffe001e4be774 fp: 0xfffffe8021317e50\n\t\t lr: 0xfffffe001d4ea934 fp: 0xfffffe8021317ea0\n\t\t lr: 0xfffffe001d4e6b54 fp: 0xfffffe8021317ee0\n\t\t lr: 0xfffffe001d4e779c fp: 0xfffffe8021317f20\n\t\t lr: 0xfffffe001cd54e98 fp: 0x0000000000000000\n Kernel Extensions in backtrace:\n com.apple.driver.AppleT6000PCIeC(1.0)[D6E00E5A-7BC8-33A5-9CDC-C4CF13DB1C01]@0xfffffe001e4a2250->0xfffffe001e4c7fa3\n dependency: com.apple.driver.AppleARMPlatform(1.0.2)[F8A12C7A-9C6E-3DCC-A2C3-56A2050A7E73]@0xfffffe001d78def0->0xfffffe001d7dc45f\n dependency: com.apple.driver.AppleEmbeddedPCIE(1)[2BA5358A-87CA-33DF-A148-FFE0C2733367]@0xfffffe001ddc8400->0xfffffe001dde2817\n dependency: com.apple.driver.ApplePIODMA(1)[7CB8682A-FA16-3841-9E30-B6032E5C6925]@0xfffffe001e1d7e60->0xfffffe001e1dc5eb\n dependency: com.apple.driver.IODARTFamily(1)[5C3EEB18-7785-328B-B994-3B0DA5B7D2FE]@0xfffffe001edfd870->0xfffffe001ee1135f\n dependency: com.apple.iokit.IOPCIFamily(2.9)[D6265950-5027-3125-A532-DC325E6A0639]@0xfffffe001f176220->0xfffffe001f1a12a3\n dependency: com.apple.iokit.IOReportFamily(47)[ED601736-6073-3B7D-A228-0FEC344992FA]@0xfffffe001f1a4ee0->0xfffffe001f1a7ecb\n dependency: com.apple.iokit.IOThunderboltFamily(9.3.3)[4F558D51-13A7-3E87-AAD6-23E0A7C0A146]@0xfffffe001f2a02c0->0xfffffe001f3dbd4f\n\nlast started kext at 341403491: com.apple.UVCService\t1 (addr 0xfffffe001c13a690, size 1772)\nloaded Presumably our kext is guilty of something, but it isn't (obviously) involved yet. Machine generally boots without our kext, but not always. Same kext works on Monterey, as well as Ventura on M1. Any particular areas to look at for this kind of panic? Are there hints in the stack? (can we resolve panic stack yet?)

The amr64 panic logs are new and a bit different, has a whole bunch of information which is nice but, sometimes I get something like: panic(cpu 11 caller 0xfffffe0013d81f1c): Kernel data abort. at pc 0xfffffe001512adb4, lr 0xfffffe001512ad9c Debugger message: panic\n Memory ID: 0x6\n OS release type: User\n OS version: 21G115\n Kernel version: Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2\/RELEASE_ARM64_T6000\n Fileset Kernelcache UUID: 39A7E336B0FAA0022B3764E49DFF29D2\n Kernel UUID: 778CC57A-CF0B-3D35-8EE8-5035142D0177\ni Boot version: iBoot-7459.141.1\n secure boot?: YES\n Paniclog version: 13\n KernelCache slide: 0x000000000bc48000\n KernelCache base: 0xfffffe0012c4c000\n Kernel slide: 0x000000000c40c000\n Kernel text base: 0xfffffe0013410000\n Kernel text exec slide: 0x000000000c4f4000\n Kernel text exec base: 0xfffffe00134f8000 ktrace: 0xfffffe180eaaea80, tid: 144477\n\t\t lr: 0xfffffe0013551400 fp: 0xfffffe180eaaeaf0\n\t\t lr: 0xfffffe00135510c8 fp: 0xfffffe180eaaeb60\n\t\t lr: 0xfffffe001369733c fp: 0xfffffe180eaaeb80\n\t\t lr: 0xfffffe00136890cc fp: 0xfffffe180eaaebf0\n\t\t lr: 0xfffffe0013686cb0 fp: 0xfffffe180eaaecb0\n\t\t lr: 0xfffffe00134ff7f8 fp: 0xfffffe180eaaecc0\n\t\t lr: 0xfffffe0013550d4c fp: 0xfffffe180eaaf060\n\t\t lr: 0xfffffe0013550d4c fp: 0xfffffe180eaaf0d0\n\t\t lr: 0xfffffe0013d7954c fp: 0xfffffe180eaaf0f0\n\t\t lr: 0xfffffe0013d81f1c fp: 0xfffffe180eaaf270\n\t\t lr: 0xfffffe0013688ecc fp: 0xfffffe180eaaf2e0\n\t\t lr: 0xfffffe0013686fb4 fp: 0xfffffe180eaaf3a0\n\t\t lr: 0xfffffe00134ff7f8 fp: 0xfffffe180eaaf3b0\n\t\t lr: 0xfffffe001512ad9c fp: 0xfffffe180eaaf740\n\t\t lr: 0xfffffe001515ac20 fp: 0xfffffe180eaaf7a0\n\t\t lr: 0xfffffe001511a03c fp: 0xfffffe180eaaf9a0\n\t\t lr: 0xfffffe001511dc78 fp: 0xfffffe180eaafa10\n\t\t lr: 0xfffffe0015148d14 fp: 0xfffffe180eaafa40\n\t\t lr: 0xfffffe00137b8b24 fp: 0xfffffe180eaafad0\n\t\t lr: 0xfffffe0015145c4c fp: 0xfffffe180eaafce0\n\t\t lr: 0xfffffe00137cc864 fp: 0xfffffe180eaafd20\n\t\t lr: 0xfffffe00137b88c8 fp: 0xfffffe180eaafda0\n\t\t lr: 0xfffffe00137cc7ac fp: 0xfffffe180eaafdb0\n\t\t lr: 0xfffffe0013bbaa28 fp: 0xfffffe180eaafe50\n\t\t lr: 0xfffffe0013686d84 fp: 0xfffffe180eaaff10\n\t\t lr: 0xfffffe00134ff7f8 fp: 0xfffffe180eaaff20\n Kernel Extensions in backtrace:\n com.apple.filesystems.hfs.kext(583.100.10)[45F25204-8A60-3A88-B71F-974BDDBDB3BF]@0xfffffe00151148a0->0xfffffe00151634e3\n dependency: com.apple.filesystems.hfs.encodings.kext(1)[4183166A-286A-3CEB-8C2C-AF85AA1F4D16]@0xfffffe00151634f0->0xfffffe001516441f\n\n last started kext at 3074954554: com.apple.filesystems.smbfs\t4.0 (addr 0xfffffe00133f4c30, size 65195)\n loaded kexts:\n org.openzfsonosx.zfs\t2.1.99\n com.apple.filesystems.smbfs\t So if you are really lucky, it will list the address of your kext here, in this case, just com.apple.filesystems.hfs.kext. But nearly all the time, you have no way to get the load address for org.openzfsonosx.zfs, which I think means I can not lookup symbols, or anything useful at all. I think HFS called into ZFS and we returned something cursed. Would it be possible to have the load addresses listed in the large list of kext loaded?

Not really a question. As part of porting other platform code, FreeBSD and Linux, there is a #define macro used to specify module parameters. It is desirable for these new sysctl to show automatically when "upstream" adds them. (without having to manually maintain a list) This is usually done with "Linker Sets" but they are not available in kexts, mostly due to __mh_execute_header. I took a different approach with: #define ZFS_MODULE_PARAM(scope_prefix, name_prefix, name, type, perm, desc) \ SYSCTL_DECL( _kstat_zfs_darwin_tunable_ ## scope_prefix); \ SYSCTL_##type( _kstat_zfs_darwin_tunable_ ## scope_prefix, OID_AUTO, name, perm, \ &name_prefix ## name, 0, desc) ; \ __attribute__((constructor)) void \ _zcnst_sysctl__kstat_zfs_darwin_tunable_ ## scope_prefix ## _ ## name (void) \ { \ sysctl_register_oid(&sysctl__kstat_zfs_darwin_tunable_ ## scope_prefix ## _ ## name ); \ } \ __attribute__((destructor)) void \ _zdest_sysctl__kstat_zfs_darwin_tunable_ ## scope_prefix ## _ ## name (void) \ { \ sysctl_unregister_oid(&sysctl__kstat_zfs_darwin_tunable_ ## scope_prefix ## _ ## name ); \ } Ie, when macro is used, I use __attribute__((constructor)) on a function named after the sysctl, which is then called automatically on kext load, and each one of those functions, call sysctl_register_oid(). And likewise for destructor / unregister. So far it works quite well. Any known drawbacks? I've not tested it on M1.

Having a peculiar issue trying to support the use of O_EXCL. (Fail if O_CREAT and file exists). It will fail the first time, then if the call is repeated, it works as expected. It is not entirely clear how macOS should handle O_EXCL, it has been mentioned that vnop_create() should always return EEXIST - does that mean even in the success case, it should return EEXIST instead of 0? That seems odd. Output of test program is: # (1) Create the file with (O_WRONLY|O_CREAT). open okay write okay close okay 86 -rw-r----- 1 501 0 29 Jan 12 17:08 /Volumes/BOOM/teest.out Deleting /Volumes/BOOM/teest.out # (2) Try creating with (O_WRONLY|O_CREAT|O_EXCL). writef: Stale NFS file handle 436207628 87 ---------- 1 501 wheel 0 0 "Jul 9 07:53:53 2037" "Jan 12 17:09:02 2022" "Jan 12 17:09:02 2022" "Jan 1 09:00:00 1970" 1048576 0 0 /Volumes/BOOM/teest.out So, since the file is deleted in between the tests, O_EXCL shouldn't really kick in here, and yet, something goes wrong. The nfs server sends ESTALE to the nfs client. The dtrace stack is: Stack: kernel.development`nfsrv_setattr+0x7c6 kernel.development`nfssvc_nfsd+0xbdc kernel.development`nfssvc+0x106 kernel.development`unix_syscall64+0x2ba kernel.development`hndl_unix_scall64+0x16 Result: 0 259014 nfsrv_setattr: entry 0 259014 mac_vnode_check_open:entry 0 259015 hook_vnode_check_open:return 2 nfsd 0 259015 mac_vnode_check_open:return 2 nfsd 0 229396 nfsrv_rephead:entry 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 0: 46 00 00 00 F... So, nfssrv_setattr() replies with 0x46/70 (ESTALE) seemingly because the call hook_vnode_check_open() returns 2 (ENOENT). Why though, the file was removed, I verified the cache has no entry. Then created again, confirmed it IS in the cache. <zfs`zfs_vnop_remove (zfs_vnops_osx.c:1700)> zfs_vnop_remove error 0: checking cache: NOTFOUND <zfs`zfs_vnop_create (zfs_vnops_osx.c:1427)> *** zfs_vnop_create: with 1: EXCL <zfs`zfs_create (zfs_vnops_os.c:660)> zfs_create: zp is here 0x0 <zfs`zfs_vnop_create (zfs_vnops_osx.c:1458)> ** zfs_vnop_create created id 82 <zfs`zfs_vnop_create (zfs_vnops_osx.c:1475)> zfs_vnop_create error -1: checking cache: FOUND I am having issues finding where the code for hook_vnode_check_open comes from anyway? The failure call in nfs server is: if (!error && mac_vnode_check_open(ctx, vp, FREAD | FWRITE)) { error = ESTALE; } So uh, why? If I let the test run again, this time the file exists, it returns EEXIST as expected. If I run the first test twice, ie, without O_EXCL, both work. So it seems to only go wrong with O_EXCL, and file doesn't exist. It is curious as to why nfs server figures out that exclusive is set, then clears va_mode? case NFS_CREATE_EXCLUSIVE: exclusive_flag = 1; if (vp == NULL) { VATTR_SET(vap, va_mode, 0); But doesn't use exclusive_flag until after calling VNOP_CREATE(), and it doesn't pass it either.