Hi all, I followed Modernize PCI and SCSI drivers with DriverKit and the source to implement MSI interrupts. Right now, if I first load KEXT, then completely remove it and load DEXT without rebooting the storage, DEXT interrupts work fine. But if I restart the storage, DEXT interrupts stop working. Here is the implementation for initializing MSI-based Interrupts. // MSI-based interrupt kern_return_t DRV_MAIN_CLASS_NAME::InitializeInterrupt() { // // please refer to https://developer.apple.com/forums/thread/779842 // and // https://github.com/apple-oss-distributions/IOPCIFamily/blob/main/PEX8733/PCIDriverKitPEX8733/PCIDriverKitPEX8733.cpp // Log("InitializeInterrupt() - Start"); kern_return_t result = kIOReturnSuccess; uint64_t interruptType = 0; uint32_t msiInterruptIndex = 0; // Interrupt Intialize And Setup while((result = IOInterruptDispatchSource::GetInterruptType( ivars->pciDevice, msiInterruptIndex, &interruptType)) == kIOReturnSuccess) { if ((interruptType & kIOInterruptTypePCIMessaged) != 0) break; // 0x00010000 msiInterruptIndex++; } Log("MSI index: %u, type: 0x%llx", msiInterruptIndex, interruptType); __Require_Action( result == kIOReturnSuccess, Exit, LogErr("IOInterruptDispatchSource::GetInterruptType result 0x%08x", result)); uint16_t msiControl; ivars->pciDevice->ConfigurationRead16(0x52, &msiControl); Log("MSI Control Register: 0x%04x", msiControl); // uint64_t msiOffset; result = ivars->pciDevice->FindPCICapability(kIOPCICapabilityIDMSI, 0, &msiOffset); if (result == kIOReturnSuccess) { Log("MSI Capability Offset: 0x%llx", msiOffset); } else { LogErr("Failed to find MSI capability, error: 0x%x", result); } uint32_t msiAddr; ivars->pciDevice->ConfigurationRead32(msiOffset + 0x4, &msiAddr); uint32_t msiData; ivars->pciDevice->ConfigurationRead32(msiOffset + 0xC, &msiData); Log("MSI Address: 0x%x, MSI Data: 0x%x", msiAddr, msiData); // Create interrupt queue result = IODispatchQueue::Create("InterruptQueue", kIODispatchQueueReentrant, 0, &ivars->fInterruptQueue); __Require_Action(ivars->fInterruptQueue != nullptr && result == kIOReturnSuccess, Exit, LogErr("Create InterruptQueue fail")); result = SetDispatchQueue("InterruptQueue", ivars->fInterruptQueue); __Require_Action(result == kIOReturnSuccess, Exit, LogErr("SetDispatchQueue() fail 0x%0x", result)); // Create interrupt source result = IOInterruptDispatchSource::Create(ivars->pciDevice, msiInterruptIndex, ivars->fInterruptQueue, &ivars->fInterruptSource); __Require_Action(result == kIOReturnSuccess, Exit, LogErr("IOInterruptDispatchSource fail 0x%08x", result)); result = CreateActionInterruptOccurred(sizeof(void*), &ivars->interruptOccurredAction); __Require_Action(result == kIOReturnSuccess, Exit, LogErr("CreateActionInterruptOccurred fail 0x%08x", result)); result = ivars->fInterruptSource->SetHandler(ivars->interruptOccurredAction); __Require_Action(result == kIOReturnSuccess, Exit, LogErr("SetHandler fail 0x%08x", result)); result = ivars->fInterruptSource->SetEnable(true); __Require_Action(result == kIOReturnSuccess, Exit, LogErr("interruptSource SetEnable Error 0x%08x", result)); Exit: Log("InitializeInterrupt() - End"); return result; } and InterruptOccurred() void IMPL(DRV_MAIN_CLASS_NAME, InterruptOccurred) { Log("InterruptOccurred()"); .... } and the iig file class DRV_MAIN_CLASS_NAME: public IOUserSCSIParallelInterfaceController { public: ... virtual void InterruptOccurred(OSAction* action, uint64_t count, uint64_t time) TYPE(IOInterruptDispatchSource::InterruptOccurred) //PCI Dext interrupt QUEUENAME(InterruptQueue); } Did I forget something in the InitializeInterrupt implementation? Or is there another reason for this issue? Best Regards, Charles

Did "UserDoesHBAPerformDeviceManagement" return "true"? Yes, it returns TRUE, and at the same time sets ivars->bUserDoesHBAPerformDeviceManagement as true. This ensures that TargetForID_Init() waits until UserDoesHBAPerformDeviceManagement() has completed before executing AsyncEventHandler(), which then calls UserCreateTargetForID(). IMPL(DRV_MAIN_CLASS_NAME, UserDoesHBAPerformDeviceManagement) { Log("UserDoesHBAPerformDeviceManagement() - Start"); bool ret = TRUE; ivars->bUserDoesHBAPerformDeviceManagement = true; Log("UserDoesHBAPerformDeviceManagement() - End"); return ret; } If AsyncEventHandler() is not called, then UserCreateTargetForID() will not be invoked by the system. Additionally, TargetForID_Init() is called by AME_Host_RAID_Ready(), that runs inside a while loop in the UserInitializeController(). while (1) { bool sts; sts = AME_Check_RAID_Ready_Status(pAMEData); bool b_device_menagement = ivars->bUserDoesHBAPerformDeviceManagement; if (sts == TRUE && ivars->bUserDoesHBAPerformDeviceManagement) { AME_Host_RAID_Ready(); // will call TargetForID_Init() to invoke AsyncEventHandler() and then call UserCreateTargetForID() ... break; } } I suspect that what's actually going on is that CreateTargetForID is still in flight, which means you tried to reenter it by calling "UserCreateTargetForID". How can I avoid re-entering UserCreateTargetForID() and ensure that CreateTargetForID() has already been executed? Best Regards, Charles

Following is the sequence of current implementation Start() Timer_Init() UserInitializeController() In UserInitializeController(), we loop to check whether the RAID is ready and also watch a flag to see if UserDoesHBAPerformDeviceManagement() has been called, then invoke the TargetForID_Init() to call UserCreateTargetForID() Then we proceed with: Timer_Start() ZtsTimerOccurred() UserGetDMASpecification() UserReportInitiatorIdentifier() // *identifier = 65 UserReportHighestSupportedDeviceID() // *id = 63 UserReportMaximumTaskCount() // *count = 200 UserMapHBAData() // run 200 times UserStartController() UserDoesHBAPerformDeviceManagement() UserInitializeTargetForID() // runs for Target IDs 0 through 63 // Each call is paired with UserProcessParallelTask() TargetForID_Init() // be invoked in a loop process within UserInitializeController() Inside the log of TargetForID_Init() we see: TargetForID_Init() - Start AsyncEventHandler(ivars->fAsyncEventHandler, 0) !!! ERROR : Failed at LUN 0: 0xe00002d8 !!! ... Initialize Total 64 TargetForID, 0 Successfully TargetForID_Init() - End Questions: Does UserCreateTargetForID() have to be called only after UserDoesHBAPerformDeviceManagement()? What exactly does the error message 0xe00002d8 mean in this context? I never explicitly called UserInitializeTargetForID(), so why is it being executed? Looking forward to your reply. Really appreciate it! Best Regards, Charles

I figured out the reason why macOS crashed — it was because I called SetDispatchQueue() before assigning a value to the queue variable. // Create and set dispatch AuxiliaryQueue ret = IODispatchQueue::Create("AuxiliaryQueue", kIODispatchQueueReentrant, // Options (e.g., allow reentrancy) 0, // Priority &ivars->fAuxQueue); if (!ivars->fAuxQueue) LogErr("Create AuxiliaryQueue fail"); __Require((kIOReturnSuccess == ret), Exit); ret = SetDispatchQueue("AuxiliaryQueue", ivars->fAuxQueue); if (kIOReturnSuccess != ret) LogErr("SetDispatchQueue AuxiliaryQueue"); __Require((kIOReturnSuccess == ret), Exit); // Create and set dispatch InterruptQueue ret = IODispatchQueue::Create("InterruptQueue", kIODispatchQueueReentrant, 0, &ivars->fIntQueue); if (!ivars->fIntQueue) LogErr("Create InterruptQueue fail"); __Require((kIOReturnSuccess == ret), Exit); // AME_IO_milliseconds_Delay(2000); // 200ms delay #if 0 // Fail to SetDispatchQueue() ret = SetDispatchQueue("InterruptQueue", ivars->fIntQueue); if (kIOReturnSuccess != ret) LogErr("SetDispatchQueue InterruptQueue"); __Require((kIOReturnSuccess == ret), Exit); #endif However, there's still an issue: the second call to SetDispatchQueue() fails, even if I add a sleep before it. UserInitializeController() - Start !!! ERROR : SetDispatchQueue InterruptQueue !!! !!! ERROR : ret = 0xe00002c2 !!! UserInitializeController() - End Additionally, at first, UserMapHBAData() wasn't working either. The logs showed the following error: UserMapHBAData() - Start !!! ERROR : Buffer allocation failed !!! !!! ERROR : ret = 0xe00002bd !!! UserMapHBAData() - End Eventually, I was able to fix it by experimenting with different parameter values: // ivars->fTaskDataSize = 64; // ivars->fTaskDataSize = 128; // ivars->fTaskDataSize = 512; ivars->fTaskDataSize = 1024; // ivars->fTaskDataSize = 4096; const uint32_t alignment = 4; // const uint32_t alignment = 0; // Pass 0 for no guaranteed alignment. ret = IOBufferMemoryDescriptor::Create(kIOMemoryDirectionOutIn, ivars->fTaskDataSize, alignment, &buffer); if (!buffer) LogErr("Buffer allocation failed!"); Log("ivars->fTaskDataSize = %llu, alignment = %d", ivars->fTaskDataSize, alignment); __Require((kIOReturnSuccess == ret), Exit); As you mentioned, the best documentation is working sample code. If Apple could provide complete, buildable source examples, it would be an immense help, both to Apple and to developers. Lastly, you said: In the case here, I would start by simply getting the I/O ‘pipeline’ working WITHOUT actually doing any I/O. How should I go about doing that, exactly? Best Regards, Charles

How can help that anyone who comes across TYPE or OSAction also discovers a good, working example that demonstrates how to use OSAction with a timer event?

After encountering issues with the timer firing incorrectly, I refactored the code for setting up a 1-second timer in macOS DriverKit. Below is the implementation: // Use the Mach Absolute Time clock uint64_t clockType = kIOTimerClockMachAbsoluteTime; // Convert Mach Absolute Time to nanoseconds mach_timebase_info_data_t timebaseInfo; mach_timebase_info(&timebaseInfo); // Get the timebase info // Get the current Mach Absolute Time uint64_t currentTime = mach_absolute_time(); // Conversion: Mach Absolute Time * (timebase.numer / timebase.denom) uint64_t currentTimeInNanoseconds = (currentTime * timebaseInfo.numer) / timebaseInfo.denom; // Calculate the deadline: 1 second from now in nanoseconds uint64_t oneSecondInNanoseconds = NSEC_PER_SEC; // 1 second uint64_t deadlineInNanoseconds = currentTimeInNanoseconds + oneSecondInNanoseconds; // Convert back to Mach Absolute Time units uint64_t deadline = (deadlineInNanoseconds * timebaseInfo.denom) / timebaseInfo.numer; // Leeway: Set to 0 for precise timing (can adjust for power efficiency) uint64_t leeway = 0; kern_return_t ret = ivars->m_zts_timer_event_source->WakeAtTime(clockType, deadline, leeway); if (ret != kIOReturnSuccess) { LogErr("Failed to schedule WakeAtTime: 0x%x", ret); return ret; } Log("Timer successfully scheduled to wake in 1 second"); Deadline Calculation: By calculating the deadline with the proper Mach time conversion, the timer behaves as expected.

Dear Kevin, After tracing through the code, here’s my understanding of the steps involved: Initialization: In the init method, assign a work queue to m_work_queue using GetWorkQueue(). Then create a timer dispatch source (IOTimerDispatchSource) link with ZtsTimerOccurred_Impl, and associate it with the work queue. ivars->m_work_queue = GetWorkQueue(); error = IOTimerDispatchSource::Create(ivars->m_work_queue.get(), &zts_timer_event_source); ivars->m_zts_timer_event_source = OSSharedPtr(zts_timer_event_source, OSNoRetain); Next, create an action (OSAction) for handling the timer: error = CreateActionZtsTimerOccurred(sizeof(void*), &zts_timer_occurred_action); // How can this be done? ivars->m_zts_timer_occurred_action = OSSharedPtr(zts_timer_occurred_action, OSNoRetain); ivars->m_zts_timer_event_source->SetHandler(ivars->m_zts_timer_occurred_action.get()); Start I/O In the StartIO method, use DispatchSync to schedule tasks and call StartTimers(). ivars->m_work_queue->DispatchSync(^(){ ... StartTimers(); }); Start Timers Set up and enable the timer in StartTimers() to schedule an initial wake-up. ivars->m_zts_timer_event_source->WakeAtTime(kIOTimerClockMachAbsoluteTime, current_time + ivars->m_zts_host_ticks_per_buffer, 0); ivars->m_zts_timer_event_source->SetEnable(true); Timer Event Handler Handle the timer event in ZtsTimerOccurred_Impl and set the next wake-up. void SimpleAudioDevice::ZtsTimerOccurred_Impl(OSAction* action, uint64_t time) { ... ivars->m_zts_timer_event_source->WakeAtTime(kIOTimerClockMachAbsoluteTime, current_host_time + host_ticks_per_buffer, 0); ... } Question: The GetWorkQueue() & CreateActionZtsTimerOccurred methods seem to belong to the framework. How can I do if there’s no explicit API to call? In StartIO, is it necessary to use DispatchSync, or can another approach be used? Why are there two WakeAtTime calls—one in StartTimers() and another in ZtsTimerOccurred_Impl? What’s the purpose of this design? Best Regards, Charles

I think I now understand what the memoryIndex should be set to. Here’s the updated log output: !!! ERROR : Failed to get BAR0 info (error: 0xe00002f0). !!! BAR1 - MemoryIndex: 0x00000000, Size: 0x00040000, Type: 0 !!! ERROR : Failed to get BAR2 info (error: 0xe00002f0). !!! And here’s the function I used to retrieve the BAR information: IOReturn GetBARInfo(IOPCIDevice* pciDevice, uint8_t barIndex, uint8_t* memoryIndex, uint64_t *barSize, uint8_t* barType) { IOReturn ret = pciDevice->GetBARInfo(barIndex, memoryIndex, barSize, barType); if (ret == kIOReturnSuccess) { Log("BAR%d - MemoryIndex: 0x%08x, Size: 0x%08llx, Type: %u", barIndex, *memoryIndex, *barSize, *barType); } else { LogErr("Failed to get BAR%d info (error: 0x%08x).", barIndex, ret); } return ret; } And then, use the memoryIndex obtained from GetBARInfo to perform memory read/write operations.

Is there anyone who can help me delete this post? I accidentally revealed some information that shouldn't have been disclosed and need to retract and repost it. Thank you.

Is there anyone who can help me delete this post? I accidentally revealed some information that shouldn't have been disclosed and need to retract and repost it. Thank you.

Thank you so much, Kevin Elliott! I’ll definitely take the time to dig into the IOKit and Kext architecture docs you mentioned. It might take me a while to go through everything, if I have any question, I'll be back and ask for some help.

Don't say that, Smith. Valuable information about dext is almost impossible to find online, so any help you provide is incredibly significant to me. I'm truly grateful. This is my first time working on dext development, I always ask Copilot or ChatGPT for help, but their answers are mostly non-functional code, which is truly frustrating. For example, when I try to connect to this dext, their user space client code is not workable also. May I politely ask if you have a basic and functional Xcode project for both the dext and client sides that I could use as a reference?

Thanks, Smith, Thanks for your quick response. There are so few resource about DriverKit developing that I can find. So, thank you very much! I follow your suggestion, add the code for NewUserClient kern_return_t IMPL(testinfo, NewUserClient) { kern_return_t ret = kIOReturnSuccess; IOService* client = nullptr; Log("fredapp NewUserClient()"); ret = Create(this, "UserClientProperties", &client); if (ret != kIOReturnSuccess) { Log("fredapp NewUserClient() - Failed to create UserClientProperties with error: 0x%08x.", ret); goto Exit; } *userClient = OSDynamicCast(IOUserClient, client); if (*userClient == NULL) { Log("fredapp NewUserClient() - Failed to cast new client."); client->release(); ret = kIOReturnError; goto Exit; } Log("fredapp NewUserClient() - Finished."); Exit: return ret; } And change the CppUserClient ret = IOServiceGetMatchingServices(kIOMasterPortDefault, IOServiceNameMatching(dextIdentifier), &iterator); But still got the message: Failed to match to device. By the way, in CppUserClient.cpp ret = IOServiceOpen(service, mach_task_self_, kIOHIDServerConnectType, &connection); our dext is using the PCIDriverKit.framework, and include the headers #include <DriverKit/IOUserServer.h> #include <DriverKit/IOLib.h> #include <PCIDriverKit/PCIDriverKit.h> #include <DriverKit/IODMACommand.h> #include <DriverKit/IODispatchSource.h> #include <DriverKit/IOTimerDispatchSource.h> #include <DriverKit/IOInterruptDispatchSource.h> So I try to replace the kIOHIDServerConnectType to kIOPCIDeviceConnectType, but I do not know how to import the correct header file. I try to include #include <IOKit/PCIDriverKit.h> but Xcode report file not found. What's the next step to try? And I find a Github repository SamsungDS/MacVFN, in the file MacVFN/MacVFNUserClient.cpp, #include <DriverKit/IOUserServer.h> #include <DriverKit/IOLib.h> #include <DriverKit/OSData.h> #include <DriverKit/OSDictionary.h> #include <DriverKit/OSNumber.h> #include <PCIDriverKit/PCIDriverKit.h> If it is workable (because I was failed to include PCIDriverKit/PCIDriverKit.h), and I will try to download and inspect the code to find some information and try to figure out how to fix my problem. Best Regards, Charles