So, I have an important news, but let me reply in order. Preamble Yes, I used it intensively. It is, at least, a bit more easier to navigate the ioreg tree. Even if sometimes crashes. IORegistryExplorer.app I think that, in a certain way, we have finally agreed on the audio latency aspects. We are in a R&D phase, so we are experimenting latencies lower than what the market offers: 0.5 to 1ms roundtrip . But, if you want me to provide you a spec. I would tell you 1.8ms roundtrip. So, as I already mentioned in my previous reply, I don't think that OS scheduling or overall system load matter to a significant extent in this investigation, and measurements under load prove it. Clearly, the concurrent DDR access will cause jitter in read/write operation, but these seem to be inside the margins that I already considered. Now, the Plot Twist I decided to extensively log ioreg changes and console output during the issue occurrences and, completely unexpectedly, I discovered these kind of events. kernel [ACIO2:high_speed_lane.c:289] Gen2/3 link error. lane=0, error=83 kernel [ACIO2:high_speed_lane.c:289] Gen2/3 link error. lane=1, error=83 I therefore tried to improve my statistics by logging longer, and I discovered that those kinds of errors always preceded missed reading deadlines. No missed deadlines, no link errors – and vice versa. This means that the cause is much more lower level than what we expected. Perhaps, ironically, the cursed spectre of my previous job as a signal integrity engineer at CERN. The question now is, does this happen between FPGA to USB4 Controller or USB4 Controller to Mac? I don't think that I saw this high_speed_lane.c code into the open source IOPCI, but I might be wrong. Do you know something about it? I also didn't designed the PCB myself, this will be performed in the upcoming future. So, I don't really have control of what and how well things were managed there. This was just an AliExpress ADT-UT3G adapter that we adopted between a good PCIe FPGA card and the Mac (and, perhaps not insignificantly ^^, with the cable that came free of charge). I still require more time to investigate it but I noticed that, even a hard abruption of the FPGA does not produce such a kind of error. Instead, disconnecting the USB-C does. At the moment, I decided to try running with an older USB3.2 cable, which force negotiation to 20Gbps instead of 40Gbps, while still keeping the PCIe tunnelling alive. With that I did not see such issues anymore. Hence, the problem was probably due to a SI issue in the cable or between USB4 controller to USB-C connector. It’s been quite a turbulent story, but perhaps we’ve managed to find some answers. I’d like to summarise the whole strategy I used and include it in this post for future reference.

Dear Kevin, give me a bit of time to think about all your points. I just want to highlight that the numbers I provided do not change even if, for instance, I run a stress test in the meanwhile, or I run a Geekbench. I also tried to use the GPU intensively in the meanwhile. That 30-50us baseline is always there. This seems quite reasonable since the net bandwidth used on DDR by the DMA is very little in comparison to its capability (200MBps vs 200GBps). So I would not be too concerned about the HAL pipeline at the moment. If the load will be too high, then CoreAudio callbacks or whatever will probably overrun in the user application, but I do not believe this will affect the DMA transactions themselves (and the driver, which does not do actually nothing on the data, no copy). The problem here is that you didn't say "1ms", you said "(~30 to 50 μs)". 1ms is 1000μs. Similarly, the "spike" you're describing here: 30-50us is the average read duration, not the deadline. The deadline in my case is due to the 16 samples, which at 192kHz is 83us, at 48kHz 333us etc. Let's consider instead that we want 1ms, that is 192 samples. But HAL buffer has a minimum of 32, so that will not work. I would have 32,64,96,128 unavailable by design. Finally, the question concerning why not using the same size as HAL. Different reason: Paradoxically, HAL buffer size is not available until StartIO is launched. And, even in that case, its real value is available only via the io_operation (in_io_buffer_frame_size). So I would have to tune my DMA transaction while it is already operating on buffer, weird. IOUserAudioIOOperation in_io_operation, uint32_t in_io_buffer_frame_size, uint64_t in_sample_time, uint64_t in_host_time) Can I really use the same size as HAL buffer keeping synchronization ok? For this I have to do a bit my exercises. Look at this diagnostics I have set in console: default 20:29:37.205137+0200 kernel DiagnosticTimerOccurred_Impl: Host Out - HW: 48.000000, HW - Host In: 112.000000 So, here you se the HAL pointer - the DMA pointer (for outputs, that are DMA read) and the counterpart. In this example I set a 64 sample HAL buffer + 16 safety margin. 48 is less than 64, so it means that, if my DMA fires 64 sample in the upcoming future, it will probably overlap HAL pointer -> bad. So I’d like to tread a bit more carefully such an approach. If possible, I will of course implement it. ...then I believe it's capable of functioning in either mode. More to the point, I believe you're building on IOPCIDevice, which means you're using Thunderbolt, not USB. Yes the device is exactly the one you mentioned and, true, I'm building on IOPCIDevice. So ok what you said. I just want to be sure that my Power Management directives are correctly propagated to all parents and, lower level, components.

Thunderbolt over USB-C is still just... Thunderbolt. That is, the first thing that happened when the USB-C connector was plugged in was that everyone agreed to "talk Thunderbolt", at which point the entire USB-C spec was ignored and the bus became a Thunderbolt bus. Ok, that's interesting. I only want to clarify that I'm using an ASMEDIA ASM2464PDX, which is mentioned as a USB4 device (no Thunderbolt Certification Logo). I was wondering if still some link layer features apply. In practical terms, I noticed: LPM policies in the USB4 controller ioreg -l -p IOService -w 0 | grep -E "LPMPolicy|USB4LPM" | | | | | "UsbHostControllerUSB4LPMPolicy" = 1 | | | | | "UsbHostControllerUSB4LPMPolicy" = 1 | | | | | "UsbHostControllerUSB4LPMPolicy" = 1 ioreg -l -p IOService -w 0 | grep -A 20 "AppleSynopsysUSB40XHCI" | grep -E "kControllerStat|CurrentPowerState|DevicePowerState|LPMPolicy" reporting very low percentage of kPowerStateOn in my device and, remarkably, a CurrentPowerState transition which appears to have taken place in the same time window of the deadline miss. In particular 2. is probably not negligible and maybe worth further investigation. That may sound strange, but the rough math says that's only ~300 MB/s, which isn't a lot of data on Thunderbolt. True. This is, as you mentioned, the result of the high number of non-posted read operation overhead. Instead, note that the "posted" write operations are here in the ~3.8GB/s area, even if the transition rate is the same. Now let's talk about audio. However, I'm also not sure why you need to be doing ~20,000 reads/s. and the shortest detectable audio latency is normally put in the range of ~5ms[1], which translates to 200 op/s. That's FAR fewer than what 50 us would imply/require. [1] And I really do mean "shortest". "Acceptable" audio latency is significantly higher, often MUCH higher. Let's clarify this together with the latency related discussion. First, the specs that you mentioned are well exceeded by modern audio cards. Many high-end brands achieve <2ms roundtrip latency, so the problem solution exists. Is this low-latency needed? In several occasion Yes. As a musician, as well as an engineer, I can easily tell whether my sound card is running at 2 or 5 ms during a live performance. 10 ms starts to become even annoying if the monitoring / PA system is close to the performer. So, I don’t think the aim here is to question the specifications of our project. Concerning the scheduling and HAL pipeline. I see you point. The real-time scheduled driver thread is a thing and HAL thread is another, but I can guarantee after many years of experience in the field, that CoreAudio and upper level pro audio applications succeed in sustaining low latencies even in the range of ~1.5ms (that is e.g. 64 samples at 48kHz) without glitches. You might say, ‘Not at 256 channels.’ That’s probably true. But with dozens of channels, it certainly is. And the high number is down more to routing flexibility than to a need for concurrent use. So my goals are demanding but not SciFi. Furthermore, I probably was not enough clear, but in all these tests I, of course, have employed the entire HAL layer. In fact, data are checked via a custom user application or on pro-DAW working at 32 or 64 HAL buffer sizes. Never reported glitches except in the mentioned events. Having said that, I think that we all agree that larger, and less frequent, transfers would reduce read overhead, but I cannot increase them too much for the exposed reasons. I can maybe go up to 32, to try matching the minimum HAL buffer, but not higher than that. What I currently do is just matching the safety margin. #define BUFFER_SAFETY_MARGIN (16) SetInputSafetyOffset(BUFFER_SAFETY_MARGIN); SetOutputSafetyOffset(BUFFER_SAFETY_MARGIN); This comes from the fact that such a value should be, in my opinion (which can be wrong), equivalent to the synchronization uncertainty between HAL and DMA buffer pointers. Wrapping the results of our last test experiment: 256 I/O, 16 sample burst @192kHZ (a deadline of 83.3us). We report that: Read takes in average about the 42% of the deadline and the deadline miss is rare (<1e-10). Write time is negligible and stable, even in the case of spike -> Info which can be important in the investigation! Deadline misses causes not predictable high read time spikes (>350us). Throughput and overall performance comply with the project requirements. Given that I state that: We do not need te deploy actions which increase the throughput. We cannot just increase the transfer size due to latency requirement. Such increase could even not been engineered properly due to the non predictable spike read duration ( we will not have a number of the presumably safe buffering size, even 1M samples cannot be sufficient to tackle the problem with such a methodology ). So, the work focuses to the tasks: Deploy failsafe logic into the FPGA (e.g. skip samples in case of deadline misses) Understand the nature of such a spike (DART, Power Management etc.) and deploy all the features that macOS provides us to avoid / minimise it. I mean, if this system works correctly for the 99.999% of the time, there is for sure an Apple Engineer which can tell me why in that 0.001% my read takes 10 times the usual time. I'm sure the cause can be found and tackled. This is not a cosmic ray bit-flipping my DDR, it is in some way a system decision. I'm an expert in 1., but I need help for 2. providing whatever code / measurement required from your side.

Dear Kevin, I had the opportunity to upgrade my DMA to allow bursting 16KB at each read/write operation on the mentioned, 16KB aligned, buffer. This translates to transfers of 16 samples, 256 audio channels (32bit). It is not trivial to isolate performance improvement due to the larger bunch size (it was 4 or 8 samples earlier) from the lucky 16KB number itself, but it seems that the overall read operation time did not change (~30 to 50 us) even if the data amount is now doubled, so I'm happy of the result anyway. MRd to CpID takes about 15-40 us and data passing itself 5-10 us. The figure shows internal AXI transactions (write: yellow, read: cyan), which will then become PCIe TLP, in the mentioned conditions. I successfully ran the prototype for 12 hours and everything worked like a charm. Then, I decided to try using another USB-C on the MacBook, just to exclude possible routing performance discrepancies. So I moved from the single one on the right (close to the HDMI) to the left one, close to the MagSafe. Procedures went fine for 20 minutes even if read times were noticeably longer (~40-60us) but, then, I unfortunately got a 350us spike in the read, which caused over/underrun. I repeated the test and another one occurred. At the third test, baseline times went smaller and no problem occurred for the next 2 hours. I also tried a reboot, but still the same good behavior, like as a self-training machine was operating under the hood. I was not able to reproduce the problem so far. Thinking a bit about the issue, I noticed that I probably have included in my driver measures against aggressive CPU and PCIe power management, but not covering the USB4 layer at all. Have you some indication about this? Replying to the remaining questions. I clearly see that pushing my device throughput further, will force me to come to terms with user client layer and upper. From one point of view, the extrapolation of some overall numbers could be considered part of the experiment. On the other hand, everything on top of the driver is more CoreAudio / HAL people job. I think that trying to push my own system (driver/hardware) performance to the limit is still worth. Applications can eventually go beyond audio itself. Having said that, my target is just in the configuration and specs that I already mentioned, no more for the moment. Instead, talking about the non hard-realtime nature of the OS. That is completely clear. I know that the exception, or deadline miss, is just round the corner. Currently, my system is protected against single deadline miss. If, for instance, the read is not completed into the deadline, the new read remains pending and it is completed as soon as the first one finishes. This is ok but it cannot help in case more than a period is exceeded. I will implement protections which will eventually just skip packets and resume a clean stream in catastrophic conditions. But this will, and has to, be implemented in a second moment. Now, let's just trying to tackle something that is relatively uncommon, but definitely not extraordinary. I can provide any other info, test result, if required. Thank you very much for your time and support.

Thanks for the reply Kevin. My apologies for the too qualitative info. The device prototype has been just set up and I don't have enough good statistics yet. I currently would like to ensure that all the proper driver technologies have been put in place and I will then start a long run session. Audio Buffers Let me provide you more detail about the system and the tests carried out so far. I will present the methods concerning the D2H path only (the one affected by the latency spike). The write one is anyway completely equivalent. Buffer allocation (in audio device init): `` OSSharedPtr<IOBufferMemoryDescriptor> m_input_io_ring_buffer; //into ivars IOBufferMemoryDescriptor::Create(kIOMemoryDirectionIn, buffer_size_bytes, 0x4000, ivars->m_input_io_ring_buffer.attach()); `` Buffer memory mapping (in audio device StartIO): __block OSSharedPtr<IOMemoryDescriptor> input_iomd; input_iomd->CreateMapping(0, 0, 0, 0, 0, ivars->m_input_memory_map.attach()); In all tests, a 16384 audio sample buffer has been used. The total size depends on how many channels were interleaved. Particularly I tested a system with 16, 64 and 256 I/O audio channels, 48kHz, 32 bit integer format. DMA Buffer Preparation D2HSegmentsN = 1 // Single segment forced (so far) IODMACommand::Create(ivars->pciDevice, kIODMACommandCreateNoOptions, &dmaSpecification, &dmaCommandD2H); dmaCommandD2H->PrepareForDMA(kIODMACommandPrepareForDMANoOptions, D2H_memory_buffer_descriptor, 0, virtualD2HSegment.length, &mem_direction_flags, &D2HSegmentsN, physicalD2HSegment); PCIe Device Followed the same procedure presented in official Apple video for DMA bus mastering ("Modernize PCI and SCSI drivers with DriverKit"). // Enable memory space access and bus mastering for DMA ivars->pciDevice->ConfigurationRead16(kIOPCIConfigurationOffsetCommand, &commandReg); commandReg |= (kIOPCICommandBusMaster | kIOPCICommandMemorySpace); ivars->pciDevice->ConfigurationWrite16(kIOPCIConfigurationOffsetCommand, commandReg); Performed Tests Very First. No actions for CPU/DART/PCIe power management (all default), 16 Channels, single DMA burst at every audio sample (20.8us of deadline), that is 64 bytes (very inefficient). Frequent deadline misses (1 per minute) in the read operation. This is predictable since the baseline takes normally about ~20/25us -> abandoned approach. Burst increased to 8 audio samples (that is 167us of deadline) and 16 interleaved channels (512 bytes). Better stability in operation (read baseline is still about 10 to 40us). However, 1 per 30 minutes c.ca I noticed a spike in the read exceeding the deadline -> host underrun (bad). Same burst morphology but I applied power management + bus characteristic constraints. Particularly: pciDevice->EnablePCIPowerManagement(kPCIPMCSPowerStateD0); pciDevice->SetASPMState(kIOPCILinkControlASPMBitsDisabled); //This looks very critical <<<<------- RequireMaxBusStall(kIOMaxBusStall25usec); plus, into Info.plist: IOPCITunnelL1Enable NO IOPMPCISleepLinkDisable NO IOPMPCIConfigSpaceVolatile NO IOPCIRetrainLinkWake YES Now things are much better and read deadline misses occurred only probably 3 times in 12 hours test. Carried away by my enthusiasm, I tried an extreme test with 256 channels. The burst was of 8 or 4 samples, which indeed corresponds to 8KB or 4KB. The outcome seems very similar to case 3. But I’d like to eliminate the possibility of deadline misses entirely. So I went further on investigating about power features etc. I ended up adding this requirements before the audio IO op. start: ChangePowerState(kIOServicePowerCapabilityOn); SetPowerOverride(true); CreatePMAssertion(kIOServicePMAssertionCPUBit | kIOServicePMAssertionForceFullWakeupBit, &ivars->PMAssertionID, false); After this, in several days, I did not notice any relevant event and my question is if the problem has been really solved completely (?). I should probably try to comment the called method one by one and check what is the game changer. Am I doing some stupidities? Are some of these method redundant (probably yes). Are there other relevant methods I'm missing or some profile tools from the host system which I can use to track the system in long term? All the cited measurements have been carried out by the FPGA itself, so they are reliable in term of precision. Concerning your point of the 16KB, I know this is the page size, I can try to ask my DMA to produce such a burst. However, if I remember correctly, PCIe allows burst of 4KB maximum, so I don't know if this will help. I can try. Worth to study better if such a large request can be asked in a MRr, or a division In sub-chunks is unavoidable. Thank you very much