Oh huh, I *assumed* it does not fail on hfs and apfs, but turns out it does. And ssh localhost will make it not-fail. I get Operation not permitted both hfs and apfs can give chown -R error. both hfs and apfs will not give error if you ssh localhost.

Oh ah, i see. If I add "Full Disk Access" to "Terminal" (sshd already has it) then it works. That isn't all that exciting then. Ah well, on to next mystery...

Ah ok, my bad, it didnt pay attention to boot-args as I'd "cdrutil enable --without nvram" when KDK says just to disable. Not sure if either works completely, as I have no wired network for MBP yet. But the symbolication question is what I am hoping to have answered, like when users send you panic reports.

One of the users noticed something interesting; objdump -S /Library/Extensions/zfs.kext/Contents/MacOS/zfs|less 0000000000088000 _abd_verify: Using the first symbols offset, 0x88000, the stack posted in the panic makes sense. So change load address 0xfffffe002641c000 - 0x88000 = 0xFFFFFE0026394000. atos -o /Library/Extensions/zfs.kext/Contents/MacOS/zfs -arch arm64e -l 0xFFFFFE0026394000 0xfffffe00266449d4 0xfffffe002650ab60 0xfffffe002650fad4 0xfffffe002650dc88 0xfffffe0026517798 0xfffffe002763f82c getf (in zfs) (spl-vnode.c:218) zfs_file_get (in zfs) (zfs_file_os.c:412) zfs_ioc_send_new (in zfs) (zfs_ioctl.c:6472) zfsdev_ioctl_common (in zfs) (zfs_ioctl.c:7603) zfsdev_ioctl (in zfs) (zfs_ioctl_os.c:234) Similarly, lldb can be made to work with offsets - 0x88000, as loaded from 0. Not when using load address. The actual panic? Called vnode_specrdev() on a vnode not CHR/BLK.

wouldn't let me post due to "content".

It is curious as to why nfs server figures out that exclusive is set, then clears va_mode? Oh yeah, this came screaming back to me. So when NFS has to do exclusive, it attempts to create the file with metadata cleared, that is va_mode 0, and time. It uses atime to hold a create_verf (probably IP and counter), then uses getattr() to confirm the create_verf is matching (this client won). (va_flags has VA_UTIMES_NULL). If that goes well, it calls setattr() with the correct information. # We are about to create the file, with O_EXCL, no existing file. 0 nfsrv_create:entry 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd # Which finally calls our vnop_create 0 zfs_vnop_create:entry # we have created it! 0 zfs_vnop_create:return 0 nfsd # nfsrv_create now calls setattr - probably to set atime 0 zfs_vnop_setattr:entry 0 zfs_vnop_setattr:return 0 nfsd # done, and some getattr verify 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd # send reply code to client 0 nfsrv_rephead:entry 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 0: 00 00 00 00 .... # nfsrv_create returns success! We have created the file 0 nfsrv_create:return 0 nfsd # this setattr appears to be the next client request, unsure where it is from # but probably to set mode/uid/gid 0 nfsrv_setattr:entry 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd # nfsrv_setattr calls mac_vnode_check_open 0 mac_vnode_check_open:entry 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 1 zfs_vnop_getattr:return 0 nfsd # this setattr is about to fail. 1 hook_vnode_check_open:return 2 nfsd 1 hook_vnode_check_open:return 0 nfsd 1 vng_vnode_check_open:return 0 nfsd 1 mac_vnode_check_open:return 2 nfsd 1 mac_vnode_check_open:return 2 nfsd # mac_vnode_check_open failed, error = ESTALE 1 zfs_vnop_getattr:entry 1 zfs_vnop_getattr:return 0 nfsd 1 nfsrv_rephead:entry 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 0: 46 00 00 00 F... I guess if I knew where hook_vnode_check_open() is defined, I can perhaps figure out what goes wrong, but I get a bit lost in the MACF macros.

Running kernel.debug and dtrace - it looks like: 1 1 nfsrv_setattr:entry 1 1 mac_vnode_check_open:entry 1 1 mac_cred_check_enforce:return 1 nfsd 1 1 mac_label_get:return 0 nfsd 1 1 mac_error_select:return 2 nfsd 1 1 mac_error_select:return 2 nfsd 1 1 mac_error_select:return 2 nfsd 1 1 mac_policy_list_conditional_busy:return 0 nfsd 1 1 kernel.debug`nfssvc_nfsd+0x108b kernel.debug`nfssvc+0x360 kernel.debug`unix_syscall64+0x81f kernel.debug`hndl_unix_scall64+0x16 1 1 mac_vnode_check_open:return 2 nfsd 1 1 nfsrv_rephead:entry 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 0: 46 00 00 00 F... Which makes me thing that after the call to mac_label_get() it checks something and is not happy. I have not really looked into what vnode_label is but presumably has something to do with auth.

I'd be happy to not know it, if you know what I mean. But since NFS+O_EXCL isn't working right here, and yet, is working correctly with HFS, I just don't see why yet. HFS does have a handful of vnode_authorize() calls, of which I have zero. That is my current thinking anyway. There is a bunch of debug stuff in MAC, is it trivial to turn on to perhaps see why it complains? Running kernel.debug 10.15.7 Thanks for replying,

OK, dtracing the debug kernel helped here, as certain functions were not inlined: dtrace -n 'mac*:entry / execname == "nfsd" / {} ' -n 'mac*:return / execname == "nfsd" / { printf("%d %s", arg1, execname); }' -n 'nfs*:entry / execname == "nfsd" / {} ' -n 'nfs*:return / execname == "nfsd" / { printf("%d %s", arg1, execname); }' -n 'vnode*:entry / execname == "nfsd" / {} ' -n 'vnode*:return / execname == "nfsd" / { printf("%d %s", arg1, execname); }' -n 'vn*:entry / execname == "nfsd" / {} ' -n 'vn*:return / execname == "nfsd" / { printf("%d %s", arg1, execname); }' -n 'nfsrv_rephead:entry { nd = arg0 + 0x88; tracemem(nd, 4); printf("rephead %d", arg0);}' Yeah, the whole thing. Anyway, it produced this hint: 1 269114 mac_vnode_check_open:entry 1 269042 mac_cred_check_enforce:entry 1 269043 mac_cred_check_enforce:return 1 nfsd 1 232364 vn_getpath_ext_with_mntlen:entry 1 231850 build_path_with_parent:return 2 nfsd 1 232365 vn_getpath_ext_with_mntlen:return 2 nfsd 1 232364 vn_getpath_ext_with_mntlen:entry 1 231850 build_path_with_parent:return 2 nfsd 1 232365 vn_getpath_ext_with_mntlen:return 2 nfsd 1 268848 mac_error_select:entry 1 268849 mac_error_select:return 2 nfsd 1 269115 mac_vnode_check_open:return 2 nfsd ie, it's not that mac_vnode_check_open() but rather that vn_getpath_ext_with_mntlen() fails. (Technically, build_path_with_parent()). At the end of my zfs_vnop_create() - if I add the call: vnode_update_identity(*ap->a_vpp, NULL, (const char *)ap->a_cnp->cn_nameptr, ap->a_cnp->cn_namelen, 0, VNODE_UPDATE_NAME); I get a successful test run using O_EXCL over nfs, finally. I am not entirely sure why this is required, only place I call vnode_update_identity() is in vfs_vget(). Otherwise, I've "been getting away" with not calling it all this time. I assume it doesn't hurt to call it.

The KDK readme says: Note: Apple silicon doesn’t support installing the kernel and kernel extension variants from the KDK. So I think you just can't, even now, three major versions later.

Maybe you could write a program that will dlopen() and 32bit app, the intercept any interlibrary calls, libc, syscall, with glue to translate 32bit to 64bit and back. It would be a massive undertaking, with no guarantee that you wouldn't hit a brick wall at somepoint.. .. when all "they'd" have to do is compile it for 64 bit :) Might even be easier to go translate the 32bit assembler to 64bit in each segment. But that does nothing to help you with the "old" API calls, you'd still need to interpose them with glue.

With exhaustive printf debugging - since this is now the best available, we found out the old line: static char initial_default_block[16ULL*1024ULL*1024ULL] __attribute__((aligned(16384))) = { 0 }; Used to bootstrap our memory allocator, would not work on Ventura+M1+more-than-16GB ram. Ie, MacStudio 128G. Monterey on same hardware (As well as all lower M1s, and x86_64) are fine. We replaced it with: initial_default_block = IOMallocAligned(INITIAL_BLOCK_SIZE, 16384); memset(initial_default_block, 0, INITIAL_BLOCK_SIZE); Unlikely anyone else would bump into this issue, but it feels more complete to post the answer.

uint32_t getcpuid(void) { #if defined(__aarch64__) uint64_t mpidr_el1; asm volatile("mrs %0, mpidr_el1" : "=r" (mpidr_el1)); return (uint8_t)(mpidr_el1 & 0xff); #else return ((uint32_t)cpu_number()); #endif } Is a start, on my 8core M2, it returns values 0x00 to 0x03 from Aff0 bits. Generally the 4 fast cores. If you idle for a bit, you can probably look at the Aff1 (cluster) values, returning values like 0x0102 - which probably means the slower cores. But for the sake of reducing contention by spreading out locks over cores, this is a good start.

To be complete, this is what we went with: #if defined(__aarch64__) uint64_t mpidr_el1; asm volatile("mrs %0, mpidr_el1" : "=r" (mpidr_el1)); /* * To save us looking up number of eCores and pCores, we * just wrap eCores backwards from max_ncpu. * 0: [P0 P1 P2 ... Px Ex .. E2 E1 E0] : max_ncpu * * XNU: Aff2: "1" - PCORE, "0" - ECORE */ #define PCORE_BIT (1ULL << 16) if (mpidr_el1 & PCORE_BIT) return ((uint32_t)mpidr_el1 & 0xff); else return ((max_ncpus -1) - (uint32_t)(mpidr_el1 & 0xff)); #else return ((uint32_t)cpu_number()); #endif }

Possibly it can be done, but I went to beta3 instead, which has a matching KDK.