Reboot-free upgrade of an always-matched DriverKit dext

App & System Services Drivers
Blaxkies OP
Created 1w
Replies 1
Boosts 0
Views 165
Participants 2

I have a DriverKit dext that implements a virtual SCSI HBA (no physical hardware). Because a bare IOUserSCSIParallelInterfaceController has no provider to match, the bundle ships two IOKitPersonalities:

  1. Bootstrap — IOClass IOUserService, IOProviderClass IOUserResources, IOResourceMatch IOKit. In Start() it does SetProperties({NvmeOfSCSIHBA: true}) + RegisterService(), publishing itself as a nub.
  2. Controller — IOClass IOUserSCSIParallelInterfaceController, IOProviderClass IOUserService, IOPropertyMatch {NvmeOfSCSIHBA: true} — it matches the bootstrap nub.

This loads and runs correctly. The problem is upgrades. Activating a higher CFBundleVersion via OSSystemExtensionRequest.activationRequest (in-place replace) always defers the old version's termination to reboot. The new version reaches [activated enabled] but never starts; the old process keeps running until reboot. From sysextd/kernelmanagerd:

kernelmanagerd  Dext … v15 … is being replaced and cannot be terminated right away
sysextd  delegate returns Error Domain=OSSystemExtensionErrorDomain Code=101 "…is being replaced",
         assumes responsibility for old version …, keeping old version 15
sysextd  turning the responsibility for termination of …, version 15 over to delegate
         (with uninstallation at the next reboot)
sysextd  a category delegate declined to terminate extension with identifier: …
sysextd  v15 terminating_for_uninstall → terminating_for_upgrade_via_delegate

Key observation: this defers even on a fresh boot where the dext was never opened — no app/daemon ever opened the IOUserClient, no I/O, nothing attached beyond the controller↔nub match. So it does not appear to be a "client still holds it open" / busy-state situation; the driver_extension category delegate declines the moment it's a replacement.

What I've tried:

  • In-place activationRequest (replace): always defers to reboot (above).
  • deactivationRequest (standalone): the request hangs — no delegate callback at all (waited ~13 min), even with no client open.
  • Disconnecting all clients first (graceful Stop() that cancels its dispatch queues and completes async) does not change the replace deferral.

My understanding from the docs/forums is that the normal reboot-free replace relies on the backing device being disconnected/reconnected to quiesce the old dext (thread 677040). My controller matches a persistent IOUserResources-backed nub that never detaches, so there's no equivalent quiesce point.

Questions:

  1. For a dext whose only provider is a self-published IOUserResources nub (no detachable hardware), is reboot-free replacement structurally impossible — i.e. is the Code=101 "is being replaced" defer inherent to this matching pattern?
  2. Is the supported way to live-upgrade such a dext to deactivationRequest → (on .completed) → activationRequest rather than an in-place replace? If so, what makes a deactivationRequest complete in-session vs. defer to reboot for an IOUserResources-matched dext — and what would cause it to hang with no delegate callback? (Daemon's IOUserClient is closed; the controller's Stop() cancels its queues and completes.)
  3. Should the dext itself proactively tear down the published nub (e.g. terminate the bootstrap IOService) before/at upgrade so the controller detaches — or does that just re-match the still-staged old personalities and relaunch the old version?
  4. Is there a recommended pattern for a virtual (hardwareless) DriverKit HBA that needs in-field, reboot-free version updates, or is reboot genuinely required for this class of dext?

Environment: macOS 27 (Tahoe)

Answered by DTS Engineer in 896127022

So, let me actually start here:

My understanding from the docs/forums is that the normal reboot-free replace relies on the backing device being disconnected/reconnected to quiesce the old DEXT (thread 677040).

The way to understand this is that most drivers don't have an "automatic" mechanism for terminating themselves. Using the storage stack as an example, the "entire" storage stack exists as long as a device is attached, regardless of whether or not a volume is mounted. That stack is only destroyed if/when the device is unplugged, at which point the "bottom" of the stack (the part connected to actual hardware) informs its child, which then initiates "teardown".

As a side comment, I'd note that ALL of the process is essentially "optional". That is, a provider cannot ACTUALLY force its children to teardown. It tells them there's a problem and they then CHOOSE to teardown. Putting that in more concrete terms:

  • The hardware driver finds out about device removal and tells his children.

  • That message goes "up" the stack (toward user space) until it reaches all the drivers at the "top" of those driver chains.

  • That "top" (in the case of storage, this would be the IOMedia leaves) level driver tears down and, as part of that, closes his parent.

  • When a driver no longer has any active opens it then initiates its own teardown and, ultimately, closes its parent.

  • That process continues until it "stops", typically because a given parent still has children or because it simply chooses not to*.

*For example, a USB controller without any USB device attached is basically "choosing" to hang out just in case another USB device shows up which it could then create nubs for.

That process is exactly the same for a DEXT, with the slight oddity being that it's possible for the "DEXT" part (meaning the user process part you created) to go away (for example, because you crash) while the kernel process still exists (because the kernel didn't teardown properly). Typically, this happens because the DEXT didn't manage tearing down its children properly.

Moving into specifics:

  1. For a DEXT whose only provider is a self-published IOUserResources nub (no detachable hardware), is reboot-free replacement structurally impossible — i.e. is the Code=101 "is being replaced" defer inherent to this matching pattern?

It's not impossible, but it also won't happen on its own. Two paths here:

  1. You MIGHT be able to trigger unload by sending the right unload through kmutil. This would trigger unload through your kernel provider, which, in theory, would propagate to your DEXT. FYI, this was/is also possible for standard storage KEXTs; it's just rarely used because of how it confuses the bus state (how do you tell the driver to "reload" without hot plugging the device).

  2. Your DEXT unloads "itself" by directly initiating the same teardown process, typically triggered via a custom user client.

You can play around with #1, but you'll almost certainly want to implement #2, as that's the mechanism that actually makes sense in an update app.

  1. Is the supported way to live-upgrade such a DEXT to deactivationRequest → (on .completed) → activationRequest rather than an in-place replace? If so, what makes a deactivationRequest complete in-session vs. defer to reboot for an IOUserResources-matched DEXT — and what would cause it to hang with no delegate callback?

Go look in IORegistryExplorer.app. My guess is that you'll find that the storage stack never actually unloaded, which then "locks" your kernel support driver in place.

  1. Should the DEXT itself proactively tear down the published nub (e.g. terminate the bootstrap IOService) before/at upgrade so the controller detaches — or does that just re-match the still-staged old personalities and relaunch the old version?

Yes, I think you'll need to teardown all your nubs. What I'd actually do is start by disabling ALL child object creation, then test loading/unloading and then updating. I think you'll find that if you can load/unload, then you can also update. Supporting live updating then means reenabling nub creation (so your DEXT does something useful) and then implementing nub teardown (so your DEXT returns to its "start" state, which can unload).

  1. Is there a recommended pattern for a virtual (hardwareless) DriverKit HBA that needs in-field, reboot-free version updates, or is reboot genuinely required for this class of DEXT?

See above (#3).

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Replies  1
Boosts  0
Views  165
Participants  2
DTS Engineer OP
Apple
1w
Accepted Answer
Recommended

So, let me actually start here:

My understanding from the docs/forums is that the normal reboot-free replace relies on the backing device being disconnected/reconnected to quiesce the old DEXT (thread 677040).

The way to understand this is that most drivers don't have an "automatic" mechanism for terminating themselves. Using the storage stack as an example, the "entire" storage stack exists as long as a device is attached, regardless of whether or not a volume is mounted. That stack is only destroyed if/when the device is unplugged, at which point the "bottom" of the stack (the part connected to actual hardware) informs its child, which then initiates "teardown".

As a side comment, I'd note that ALL of the process is essentially "optional". That is, a provider cannot ACTUALLY force its children to teardown. It tells them there's a problem and they then CHOOSE to teardown. Putting that in more concrete terms:

  • The hardware driver finds out about device removal and tells his children.

  • That message goes "up" the stack (toward user space) until it reaches all the drivers at the "top" of those driver chains.

  • That "top" (in the case of storage, this would be the IOMedia leaves) level driver tears down and, as part of that, closes his parent.

  • When a driver no longer has any active opens it then initiates its own teardown and, ultimately, closes its parent.

  • That process continues until it "stops", typically because a given parent still has children or because it simply chooses not to*.

*For example, a USB controller without any USB device attached is basically "choosing" to hang out just in case another USB device shows up which it could then create nubs for.

That process is exactly the same for a DEXT, with the slight oddity being that it's possible for the "DEXT" part (meaning the user process part you created) to go away (for example, because you crash) while the kernel process still exists (because the kernel didn't teardown properly). Typically, this happens because the DEXT didn't manage tearing down its children properly.

Moving into specifics:

  1. For a DEXT whose only provider is a self-published IOUserResources nub (no detachable hardware), is reboot-free replacement structurally impossible — i.e. is the Code=101 "is being replaced" defer inherent to this matching pattern?

It's not impossible, but it also won't happen on its own. Two paths here:

  1. You MIGHT be able to trigger unload by sending the right unload through kmutil. This would trigger unload through your kernel provider, which, in theory, would propagate to your DEXT. FYI, this was/is also possible for standard storage KEXTs; it's just rarely used because of how it confuses the bus state (how do you tell the driver to "reload" without hot plugging the device).

  2. Your DEXT unloads "itself" by directly initiating the same teardown process, typically triggered via a custom user client.

You can play around with #1, but you'll almost certainly want to implement #2, as that's the mechanism that actually makes sense in an update app.

  1. Is the supported way to live-upgrade such a DEXT to deactivationRequest → (on .completed) → activationRequest rather than an in-place replace? If so, what makes a deactivationRequest complete in-session vs. defer to reboot for an IOUserResources-matched DEXT — and what would cause it to hang with no delegate callback?

Go look in IORegistryExplorer.app. My guess is that you'll find that the storage stack never actually unloaded, which then "locks" your kernel support driver in place.

  1. Should the DEXT itself proactively tear down the published nub (e.g. terminate the bootstrap IOService) before/at upgrade so the controller detaches — or does that just re-match the still-staged old personalities and relaunch the old version?

Yes, I think you'll need to teardown all your nubs. What I'd actually do is start by disabling ALL child object creation, then test loading/unloading and then updating. I think you'll find that if you can load/unload, then you can also update. Supporting live updating then means reenabling nub creation (so your DEXT does something useful) and then implementing nub teardown (so your DEXT returns to its "start" state, which can unload).

  1. Is there a recommended pattern for a virtual (hardwareless) DriverKit HBA that needs in-field, reboot-free version updates, or is reboot genuinely required for this class of DEXT?

See above (#3).

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Reboot-free upgrade of an always-matched DriverKit dext
First post date Last post date
 
 
Q