MacOS(Apple Silicon) IOKit driver for FPGA DMA transmission, kernel panic.

It seems that the DMA request address initiated by FPGA exceeded 32 bits, which was intercepted by PCIe root port and resulted in a kernel panic.This is also the case on macOS (M2).

Please file a bug on this, upload the full panic report there, and then post the bug number back her. I'd like to get a clearer picture of what actually failed.

I'm also a bit confused by this point:

It seems that the DMA request address initiated by FPGA exceeded 32 bits,

That address should have come from your driver, so how did it exceed 32 bits? All of our APIs for retrieving a bus address should either provide an address that meets your specification or fail entirely. More to the point, even if they some how did return a larger address, why/how would your driver have written a larger address when it was only prepared to handle a 32 bit address?

Next, the big question is how much memory are you actually trying to map? And have you done any experimentation with smaller mappings?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

filed bug number is FB21888307. I provide partial code on bug.

Reply to your question:

That address should have come from your driver, so how did it exceed 32 bits? All of our APIs for retrieving a bus address should either provide an address that meets your specification or fail entirely. More to the point, even if they some how did return a larger address, why/how would your driver have written a larger address when it was only prepared to handle a 32 bit address?

Yes, address comes from my driver by gen64IOVMSegments API. And at the beginning, I didn't know that the address allocated through the API were always >32 bit, so when I received an address >32 bit, I didn't directly return, but continued to use it, resulting in a kernel panic.

Next, the big question is how much memory are you actually trying to map? And have you done any experimentation with smaller mappings?

I trying to map 20 KB memory, I have already tried 1 Byte or 4 KB, still get 64-bit address. And maximum DMA data size would set to 8 MB, but I haven't set 8M yet.

Thanks!

I trying to map 20 KB memory, I have already tried 1 Byte or 4 KB, still get 64-bit address. And maximum DMA data size would set to 8 MB, but I haven't set 8M yet.

I haven't checked the code, but I'd expect that you'd need to be page aligned, with would be 16KB (not 4KB). I'm actually not sure why gen64IOVMSegments worked, though it's possible that using the "native" address size changed something. Ironically, I think this might have "just worked" if you'd gone straight to 8MB

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

I haven't checked the code, but I'd expect that you'd need to be page aligned, with would be 16KB (not 4KB). I'm actually not sure why gen64IOVMSegments worked, though it's possible that using the "native" address size changed something. Ironically, I think this might have "just worked" if you'd gone straight to 8MB

Let me summarize my current attempts:

IODMACommand uses 1B, 4KB, or 16KB page alignment, and allocates IOVM address >32 bits by gen64IOVMSegments(). If the DMA transfer process continues, a kernel panic will occur. PanicSting is apciec[pcic0-bridge]::handleInterrupt: Request address is greater than 32 bits.

Using gen32IOVMSegments() will return kIOReturnMessage TooLarge. If I don't set the maximum value of 8M, the result will still be the same, so I think it may not be related to the maximum value.

Based on my current attempts, I am focusing on these two issues:

1. Does Apple M silicon not allocate<=32-bit bus addresses to third-party PCIe device (FPGA)?
2. Can Apple M silicon be configured with apciec to not intercept DMA address requests with more than 32 bits?

Using gen32IOVMSegments() will return kIOReturnMessageTooLarge. If I don't set the maximum value of 8M, the result will still be the same, so I think it may not be related to the maximum value.

If you haven't already, take a look at my post here. I haven't checked the code to confirm, but I strongly suspect that calling gen32IOVMSegments with a mask of the wrong size (like 0xFFFFF000) will cause kIOReturnMessageTooLarge, as the true mask range you've specified ("0x00000000FFFFF000") is too large to fit in 32 bits.

Does Apple M silicon not allocate <=32-bit bus addresses to third-party PCIe devices (FPGA)?

Quite the opposite. We've actually gone to considerable trouble to ensure that we COULD provide 32-bit addresses to the PCI bus. Quoting "Address Translation on 64-Bit System Architectures":

"Apple solved the problem with address translation which “maps” blocks of memory into the 32-bit address space of a PCI device. In this scheme, the PCI device still sees a 4-gigabyte space, but that space can be made up of noncontiguous blocks of memory. A part of the memory controller called the DART (device address resolution table) translates between the PCI address space and the much larger main memory address space. The DART handles this by keeping a table of translations to use when mapping between the physical addresses the processor sees and the addresses the PCI device sees (called I/O addresses)."

Note that the DART is common to all of ARM machines and, in fact, many of our more recent architectural choices basically require it to exist. For example, as I talked about here, many of our drivers now assume that IODMACommand will simply never return multiple segments.

That leads to here:

Can Apple M silicon be configured with APIC to not intercept DMA address requests with more than 32 bits?

First, as background context, part of what the DART changes is that it breaks the direct link between "physical addresses" (meaning, the linear address range that represents the actual position of data in physical RAM) and "I/O addresses" (meaning, the address you put on the PCI bus). True physical addresses are effectively useless outside of the VM system (notably, you CANNOT use one on the PCI bus), which is why:

"Because there is not a one-to-one correspondence between physical addresses and I/O addresses, physical addresses obtained from the pmap layer have no purpose outside the virtual memory system itself. Drivers that use pmap calls (such as pmap_extract) to get such addresses will fail to work on systems with a DART. To prevent the use of these calls, OS X v10.3 will refuse to load a kernel extension that uses them, even on systems without a DART.

With that context, returning to here:

Can Apple M silicon be configured with apciec to not intercept DMA address requests with more than 32 bits?

Well, no, but that's because the question doesn't really make sense. As I talked about above, "I/O addresses" do not directly map to physical memory; the DART is translating all addresses that cross the bus, regardless of address size.

In any case, I think the issue here is the address masking issue I mentioned at the top.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

If you haven't already, take a look at my post here. I haven't checked the code to confirm, but I strongly suspect that calling gen32IOVMSegments with a mask of the wrong size (like 0xFFFFF000) will cause kIOReturnMessageTooLarge, as the true mask range you've specified ("0x00000000FFFFF000") is too large to fit in 32 bits.

Yes, I am currently using the inTaskWithPhysicalMask interface in my code, and the masks I have tried are 0 x0000 0000 FFFF FFFFULL，0x0000 0000 FFFF F000ULL, 0xFFFF FFFF，0xFFFF F000, and 0x0, but the result is the same, using gen32IOVMSegments will return the same error. I have also used IOBufferCacheDescriptors:: withOptions and IOBufferCacheDescriptors:: withCapacity to build memory descriptors, but the results have not changed much.

If you have time, Could you take a look at the code? I have submitted on the bug. One more thing to add: I have also tried using the prepare() interface of IODMACommand, but there have been no significant changes.

Thanks!!

First off, thank you for your patience with this. I wanted to make sure I'd gotten an authoritative answer on this, and it look a little while to get to the right person.

Next, my answer here is primarily about Apple Silicon support. I think the same code will also work on 64-bit Intel and I'm not going to try and sort out 32-bit Intel.

If you have time, Could you take a look at the code? I have submitted on the bug. One more thing to add: I have also tried using the prepare() interface of IODMACommand, but there have been no significant changes.

So, first off, what doesn't matter here is the IOMemoryDescriptor configuration or inTaskWithPhysicalMask.Your trying to map 20Kb, so I would start testing with a size of 32Kb (2 pages). You could also specify a mask of:

0x0000 0000 FFFF F000ULL (32 bits, 16Kb aligned)

...but that's simply to start with the most straightforward configuration, not because other configuration won't work fine.

What DOES matter here is the "IOMapper" argument of your IODMACommand specification:

static OSPtr<IODMACommand>
	withSpecification(SegmentFunction  outSegFunc,
	    UInt8            numAddressBits,
	    UInt64           maxSegmentSize,
	    MappingOptions   mappingOptions = kMapped,
	    UInt64           maxTransferSize = 0,
	    UInt32           alignment = 1,
	    IOMapper        *mapper = NULL,
	    void            *refCon = NULL);

Your passing in "NULL", but what you need to pass in is the IOMapper for your target I/O device, which you can retrieve by calling "IOMapper::copyMapperForDevice()".

What's going on here is that, Intel had a single global mapper and, since PPC days, IOMD physical addresses would get premapped, so "stuff worked" without driver changes, unless you specifically asked for unmapped addresses.

AppleSilicon doesn't have one DART but MANY (100+), so you need to tell IODMACommand what device you're targeting, so that the right DART does the mapping. Note that this also means that the segments returned by one IODMACommand aren't (necessarily) valid for a different device.

That also lets me explain this:

Yes, address comes from my driver by gen64IOVMSegments API. And at the beginning, I didn't know that the address allocated through the API were always >32 bit, so when I received an address >32 bit, I didn't directly return, but continued to use it, resulting in a kernel panic.

As part of the transition to AppleSilicon and increased use of DARTs, the decision was made to essentially "unmap" the bottom 32 bits of physical memory in the same way (and for similar reasons) that VM systems do. This ensure that there are no valid physical addresses below 32GB, further interfering with the kind of memory descriptor "shortcuts" that worked on Intel/PPC.

On AppleSilicon, you MUST go through IODMACommand, passing in the correct mapper to get a working I/O address.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware