BlockStorageDeviceDriverKit grant confirmed by support but shows "No Requests" in the portal. How to resolve?

Hi Kevin,

thanks a lot for the quick view and answer. You went through all the Feedback reports remarkably quickly. Huge respect for that.

Q1. Why did you choose a 4KiB maximum and 16KiB alignment for FB25095303? What were you trying to establish?

We first tried the 16KiB maximum, 16KiB alignment and one-segment configuration you had suggested. That led us to the 4KiB test.

When nothing makes sense, I sometimes like to try something seemingly pointless to confirm the pointlessness. So we stepped back into abstraction and asked where the “famous” single difference between Intel and Apple Silicon was. We focused on 4KiB versus 16KiB and tried the Intel settings on Apple Silicon. The M5 kernel’s 16KiB page size was the one thing we could not change. It was a direct comparison.

In the 4KiB diagnostic build, the host-facing maximum read and write byte counts and maximum segment byte counts were also 4KiB. Its captured IORegistry reports two HBA segments.

Q2. What happened in the 16KiB/16KiB test? Which filesystem did you format or mount and what exactly failed?

Only RAW access worked. Formatting and mounting preformatted LUNs began but then nothing happened. There was no completed filesystem mount, no error and no panic. We did not investigate where the operations stopped.

In separate tests, even exFAT caused a panic. That led us to add strict read-only mounting to the initiator. With that enabled, all the filesystems we tried mounted. Writes and formatting were rejected, including format commands issued from the command line.

Q3. How did you measure approximately 0.08 Gbit/s?

We used RAW writes, reads and byte by byte verification. The data matched and we saw no EIO or panic during the 16KiB/16KiB test. We were primarily checking for those failures. At roughly 0.08 Gbit/s, we quickly dropped that configuration.

I expected your answer about the KEXT entitlement and was pretty sure from the start. However, Micha from my team thought I should ask you anyway. Well, I won that bet. ;)

The KEXT was an experiment to answer one question: can we avoid the panic at all? With it in place, we could read, write and verify without errors or panics. Even hours-long write or read tests completed without a single EIO.

We have not yet run a dedicated test under heavy memory pressure to stress DMA and memory. At this stage, our main question was whether the approach worked at all. The results show us that we are moving in the right direction, but I agree with you that this is a useful proof rather than a solution. From my point of view, needing a KEXT just to get a DEXT working is almost an oxymoron.

That is also why we included the KEXT source with the Feedback reports. It is only meant to show how we worked around the panic in a “quick & dirty” way. Perhaps it will be useful to your team as well.

Best greetings!!

Torsten

Hi Kevin,

just for clarification:

I'm sorry, but that isn't going to happen. We're very unlikely to grant them at all today, and we've never granted them for testing purposes.

Does this mean we can get the KEXT entitlement for a rollout until everything is done on your end?

If YES and that were possible, that would be awesome.

Naturally, we wouldn't just use it for testing then and we would only need it until everything is fixed.

Best greetings,

Torsten

So, in the process of writing up a response, I ended up getting pulled into a bit of a side chain which ended up invalidating everything I'd written.

The bottom line is that I think you (and I) have been looking at this a bit wrong. The basic assumption here has always been that single-page I/O was inherently a serious performance issue (which is still probably somewhat true) that doesn't "work,” but the problem is that it turns out that one of our drivers is actually doing I/O with an individual transfer size of 4k and an overall configuration VERY similar to what yours should be.

Here are the key parameters I just found looking at the NVME controller of the MacBook Pro I'm currently writing this on:

kIOMaximumSegmentByteCount = 0x1000 -> 4096 bytes
kIOMaximumSegmentCount = 0x100 -> 256 segments
kIOMaximumSegmentByteCount * kIOMaximumSegmentCount = 1,048,576 bytes

IOMaximumByteCount = 0x100000 = 1,048,576 bytes

Now, the in-kernel NVMe controller is obviously very different than your situation, but it's not THAT different. I don't know if the performance will be good enough to ship, but there's certainly no reason why it shouldn't at least "work.”

With that context, looking at the sysdiagnoses you sent with your bug reports, I think there are serious issues with your registry configuration. You've got this set:

"IOMaximumByteCountRead" = 2080768
"IOMaximumSegmentByteCountWrite" = 2080768

...and I think that's just "wrong.” I don't think you should/need to set IOMaximumByteCountRead at all, but I think your basic parameters should be:

kIOMaximumSegmentByteCount = 0x4000 -> 16,384 bytes (1 page)
kIOMaximumSegmentCount = 128 (your current size, but it could be larger)

Which would then make:

IOMaximumByteCount = 0x4000 * 128 = 2,097,152 = 2048k

Note that I'd still recommend NOT setting IOMaximumByteCount. It's not necessary and really only has one valid value (the calculation above), which the system will take care of for you.

A few other things to look at here:

• I haven't seen a registry log that includes the entire IOMedia stack, but I would strongly recommend that you go through that stack, looking at all the keys that control I/O request management and verify that they "make sense.” I don't think they do, and I think that's where things are breaking down.

• On the testing/validation side, I think I would try opening the buffered device node ("disk,” not "rdisk") and trying to do "randomly" sized I/O "all over" the disk. IF your device configuration is set up properly, that should work fine, as the system will generate block-aligned requests and then return the data segment you asked for. If it doesn't work, then that indicates that your device configuration isn't right. FYI, I think this could be a problem:

"IOMinimumSegmentAlignmentByteCount" = 4

• One other thing to try here would be to increase the logical block size you present to 16k. I don't think that's actually required, but it is the most direct way to eliminate all unaligned I/O.

Related to all this:

In separate tests, even exFAT caused a panic.

and

Only RAW access worked.

What did the panic actually show? One thing to understand here is that, as far as IOKit is concerned, and your driver in particular, there isn’t a difference between the raw I/O and the buffered I/O the system performs. It's all just blocks that you're shuffling about. That means any difference you're seeing is caused by the VFS layer sending different requests than "you" did.

Note that this:

Formatting and mounting preformatted LUNs began but then nothing happened.

...is what we've seen happen before when a request reaches IOUserSCSIController that it's unable to generate a segment pointer for. The whole "point" of your driver's configuration should be ensuring that does not happen.

That leads to here:

We used RAW writes, reads, and byte-by-byte verification.

Were those single-page, 16k writes? Or did you also test large writes? I think what might be going on here is that your device configuration isn't actually constraining your I/O like you need it to, but your RAW tests only sent requests that fell within your constraints. You'll then blow up as soon as your requests move outside of those constraints.

Similarly, my guess is that if you run this same driver on Intel, you'll find that you’re getting larger requests than you "should.” That's exactly the same problem; it's just not failing the way it would on Apple Silicon.

Does this mean we can get the KEXT entitlement for a rollout until everything is done on your end?

No.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

This is a little difficult at the moment because we cannot take any measurements on the Apple Silicon machine until the end of October. My colleague is currently on vacation in Brazil, visiting his in-laws. However, I’ll try to reconstruct what happened from memory.

We used exactly the values you suggested. This resulted in extremely slow performance, and only RAW access worked. Mounting the filesystem was not possible. Yes, we tried the 16 KiB configuration, but that did not get us any further. If it helps, I can send you a complete log showing exactly what happens, where, how, and when, but that will take a few days.

There is also the maximum total read/write transfer size to consider. Explaining this in detail here in the forum would mean giving away a technical advantage, so if anything is unclear, please ask me to clarify.

When the data passes through the kernel, Apple adds a header. It is small, but accounting for it is extremely important. Interestingly, it differs between Intel and Apple Silicon. So yes, we can use the system values, but doing so produces large numbers of errors or, depending on the configuration, read/write or completion errors. The kernel also frequently logs EIO errors. There is no panic, but we then see SCSI errors—sometimes just a few, sometimes a large number. There is no additional logging of this except at one specific point when an error occurs.

That is why simply choosing 1 MiB or 2 MiB does not work reliably. Some additional calculations are necessary to make this work correctly and consistently.

Does this help for now?

Best greetings,

Torsten

Hi Kevin,

we have to accept the “no” regarding KEXTs, but at the moment there is no viable path forward for iSCSI on Apple Silicon. After a brief internal discussion, we have decided to halt development and take it off the table until this works without KEXT.

Personally, I don’t see a way forward for at least the next six to nine months, so further development makes no sense at this point. Even the iSCSI mathematics involved are completely different with and without a KEXT.

Nevertheless, thank you very much for your incredible support!

Best greetings!!

Torsten

We used exactly the values you suggested. This resulted in extremely slow performance, and only RAW access worked. Mounting the filesystem was not possible. Yes, we tried the 16 KiB configuration, but that did not get us any further.

OK, but understand that the only values I've actually seen were simply wrong on both Intel and Apple Silicon. They wouldn't have directly failed on Intel but would mean that the system was not working as intended.

When the data passes through the kernel, Apple adds a header. It is small, but accounting for it is extremely important.

I'm not sure what you're talking about here. If you're comparing direct I/O (through the rdisk node) to file system I/O, then you're right that there isn't a one-to-one correspondence between the direct reads a user initiates and what the file system actually sends to disk. That's partly tied to the file system’s own structure and partly caused by how read caching works in the UBC. However, that's not really a "header“; it's just doing extra I/O.

Interestingly, it differs between Intel and Apple Silicon.

I've never looked at it in detail, but, yes, I'd expect the larger page size to alter the details of exactly what I/O the UBC issues.

So yes, we can use the system values, but doing so produces large numbers of errors or, depending on the configuration, read/write or completion errors. The kernel also frequently logs EIO errors.

I believe EIO is what would result from us failing to generate fBufferIOVMAddr, however, your I/O configuration should be structured such that you never receive an I/O larger than 16k, so wiring can't fail. One easy test here— have you tried validating your Apple Silicon configuration on Intel? You can then run every possible test, and if you ever receive a request larger than 16k, you'll then know you've got a problem.

We have to accept the “no” regarding KEXTs, but at the moment there is no viable path forward for iSCSI on Apple Silicon.

Yes, I agree with that.

Personally, I don’t see a way forward for at least the next six to nine months, so further development makes no sense at this point.

Please keep an eye on this thread, as I'll update it as soon as anything changes.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

thank you for the detailed reply. I think we need to separate Apple Silicon from Intel.

On Apple Silicon, I agree with you. We tried many configurations while trying to make filesystem mounting work with the DEXT alone. Those settings were largely experimental. As long as a filesystem mount or subsequent I/O can trigger a kernel panic, further iSCSI tuning makes little sense. A reliable filesystem mount has to come first.

On Intel, I disagree that the current implementation is not working as intended. Our initiator mounts filesystems and has passed byte-exact data integrity and memory-pressure tests without EIO or kernel panics. The current benchmark reaches about 16 Gbit/s. We could pursue more throughput, but the extra CPU load would hurt other work on the Mac. For Intel NAS use, this is a balance we’re happy with.

When I used the word “header,” I meant the extra room we observed at the Intel transfer boundary inside the kernel, not additional filesystem I/O generated by the UBC. We did not measure or trace precisely where that margin comes from, since we cannot change that kernel behavior. Possible contributors include page alignment and padding, I/O mapping constraints, network hardware alignment, or bookkeeping associated with mbufs and memory descriptors. Ignoring the margin can cause errors or a sharp drop in throughput, so our Intel configuration accounts for it. In summary: “We don’t know what it is, but we accept it.” ;)

Anyway, here’s my proposed deal:

If you and your team can help make filesystem mounting and subsequent I/O reliable through the DEXT alone on Apple Silicon, without kernel panics, we’ll take care of getting the iSCSI side right. My personal goal is 105 Gbit/s. Our quick-and-dirty KEXT-assisted prototype already reached 81 Gbit/s in the RAW benchmark, so I’m curious to see what a reliable DEXT-only implementation can achieve. Then we can happily argue about geometry again. ;)

Please keep us posted. As long as a matching KDK is available for the macOS build, we’d be happy to help you and your team at any time.

Best greetings,

Torsten

Hi Kevin,

I have a small correction for you because it was bothering me and would not let me sleep. I have no idea what went wrong there.

You were absolutely right. If we align it 100% according to your specifications, a mount is indeed possible, regardless of the filesystem. However, as you also mentioned, we are light years away from iSCSI performance with this.

Read speeds are around 30 MiB/s and write speeds are around 26 MiB/s.

So these iSCSI speeds are roughly 20x faster than USB 1.1, at least that is something, but still slower than USB 2.0. That is definitely not even a start ;)

Best regards,

Torsten

You were absolutely right. If we align it 100% according to your specifications, a mount is indeed possible, regardless of the filesystem. However, as you also mentioned, we are light years away from iSCSI performance with this.

OK, good.

I think I made this clear earlier, but my intention here has been to get you to a reliable, working implementation, NOT one that was fast enough to actually be useful. That working implementation lets you validate and test as much as you possibly can "now,” which should allow you to update and very quickly ship a working solution as soon as we provide a software I/O flow. I'm sorry that I can't be more specific than that, and I appreciate your understanding and patience.

I didn't expect a fast implementation to be possible, and the fact that the Intel approach works the way it does was a surprise to me.

So these iSCSI speeds are roughly 20x faster than USB 1.1, at least that is something, but still slower than USB 2.0.

Ironically, I'm old enough to remember when this:

Read speeds are around 30 MiB/s and write speeds are around 26 MiB/s.

...was very respectable SCSI performance, back in the good old days when any "real" Mac had a stack of SCSI drives piled next to it.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware