PCI DEXT crash panics the host: IOPCIBridge::childClientCrashRecoveryGated dereferences a NULL Expansion ROM range (FB25117710)

App & System Services Drivers
baikuken OP
Created
Replies 1
Boosts 0
Views 416
Participants 2

When a PCI DriverKit extension terminates abnormally while it has its IOPCIDevice open, IOPCIFamily's child crash recovery can panic the whole host. This happens on devices whose nub publishes an Expansion ROM entry in IODeviceMemory, which is typical of GPUs and common on other PCIe cards.

What happens

IOPCIBridge::childClientCrashRecoveryGated walks the nub's IODeviceMemory array to restore BARs. For each descriptor it computes barIndex from the register number. For the ROM (config offset 0x30) that index is 8, kIOPCIRangeExpansionROM. The function then reads child->reserved->configEntry->ranges[barIndex]->start without checking for NULL. In our case ranges[8] is NULL, and the kernel takes a data abort at FAR 0.

In IOPCIFamily-726.0.5 (IOPCIBridge.cpp, around lines 7055–7085), the barIndex == kIOPCIRangeExpansionROM check comes after the dereference:

uint8_t barIndex = (addressSpace.s.registerNum - kIOPCIConfigBaseAddress0) >> 2;   // 0x30 -> 8
IOPCIRange *range = child->reserved->configEntry->ranges[barIndex];                // NULL here
uint32_t bar = child->configRead32(addressSpace.s.registerNum) & ~0xF;
if (bar != (range->start & 0xFFFFFFFF))                                            // faults
...
if (barIndex == kIOPCIRangeExpansionROM || !(kIOPCIRangeFlagBar64 & range->flags)) // too late

Environment

  • Mac Studio (M4 Max), macOS 26.7.1 (25G241), IOPCIFamily 2.9
  • PCIe GPU in a Thunderbolt 5 eGPU enclosure, third-party PCI DEXT (development-signed)
  • The nub's IODeviceMemory has four entries, the last a 128 KB Expansion ROM

Panic signature

Two panics, both with the same signature. The panicked task is the DEXT, and IOPCIFamily is the only kext in the backtrace.

Kernel data abort, ESR 0x96000006, FAR 0x0, pc = IOPCIFamily + 0x19b5c
x2 = 0x30 (ROM config offset), x23 = 0x8 (barIndex), x27 = 0x0 (range)
faulting instruction: ldr w2, [x27]

Questions

  1. Has anyone else seen host panics in crash recovery after a PCI DEXT died? Other projects document "don't kill the driver process, or crash recovery may panic the machine," and this may be the same root cause.
  2. Is there a supported way for a DEXT to avoid this? For example, can a driver keep the ROM descriptor out of the recovery path, or is avoiding DEXT crashes the only mitigation until a fix ships?

We filed this as FB25117710. If you hit the same panic, please reference that number in your own Feedback so the reports can be linked.

Replies  1
Boosts 0
Views  416
Participants  2
DTS Engineer OP
Apple

So, I wanted to start by commenting on this:

Other projects document "don't kill the driver process, or crash recovery may panic the machine," and this may be the same root cause.

DriverKit's goal/role in preventing kernel panic is more complex and nuanced than "a DEXT should never trigger panics. Its most direct goal was to remove a driver’s ability to disrupt the larger kernel, and that's basically a complete success. For example, there simply isn't any way for a PCI DEXT to trigger a kernel panic in the USB or BSD layers in the way memory management bugs in a KEXT could. Note that this is more valuable than it seems, as it helps prevent a class of bugs that were MOST difficult to investigate and debug.

However, preventing the DEXT ITSELF from triggering panics in components it directly interacts with is a more complicated issue. It's certainly a goal in the aspirational sense, but it's definitely not something we've achieved today.

Particularly in the case of PCI, I'm not sure it's ACTUALLY possible to create an API that gives a PCI DEXT the level of DMA access and performance it requires while also ensuring that API can never bring the machine down.

The bottom line here is that while DEXT developers should file bugs about any panics they find, they also shouldn't assume it can't happen or be surprised when it does.

We filed this as FB25117710.

Thank you for that. It's a well-written bug, and it's already with the right engineer.

Has anyone else seen host panics in crash recovery after a PCI DEXT died?

I haven't seen this particular panic before, but I've certainly seen DEXT panics in the past.

Is there a supported way for a DEXT to avoid this? For example, can a driver keep the ROM descriptor out of the recovery path, or is avoiding DEXT crashes the only mitigation until a fix ships?

I'm not sure. I don't see any obvious option, but I've reached out to the PCI team to see if they have any ideas.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware