fseventsd cleanup cursor exhaustion and notification stalls on macOS 26.7.1

Feedback ID: FB25090184

I've been tracking recurring fseventsd memory growth and slow file-change notifications on macOS 26.7.1 (25G241), with massive Bazel build trees. There seems to be a repeatable boundary inside event-table cleanup. I may be missing another reset path, but the live observations match the disassembly closely.

In fseventsd 1413.160.2, ARM64E UUID CDD7E21B-7F60-3DD4-AA03-8B36745DC0F4, cleanup at RVA 0x4604 uses a 32-bit scan cursor at table+0x14. The guard at 0x464c/0x4650 skips scanning above 0xfffffdff. Each complete pass advances by 512. Bucket indexing masks the cursor, but the stored cursor keeps increasing. Starting at zero, it stops at 0xfffffe00 after 8388607 complete passes.

Table resize resets the cursor. Two observed October 3 episodes reached exactly that value, accumulated zero-reference records, and recovered without a restart when the table doubled. The resize loop rehashes records rather than freeing them, so resumed cleanup after the reset seems to explain the recovery.

The reviewed resize limit is 262144 buckets. On October 6, after the table reached that size, the next cursor exhaustion persisted. Physical footprint grew from 26.6 MB to 2.31 GB in about six minutes, and fresh FSEvents probes reached an 8-second timeout. A bounded walk found 199944 zero-reference records among 200000 visited. The walk was partial and unsuspended, with linkage errors; that is not a whole-heap count.

Restarting only fseventsd restored advancing cleanup, approximately 5 MB footprint and millisecond probes. The client mix changed after restart, so this is not a controlled fixed-client comparison. Finder stayed running, and Desktop add/delete checks worked on both sides. An earlier stale-Desktop symptom may be a separate issue.

The boundary and table-size prediction were recorded before the live observations. The Feedback report has the timeline, graphs, counter snapshots, record walks, stack excerpts, disassembly and read-only sampler source. No daemon binary or target memory was changed. Live counter access used a temporary SIP debugging exception; a severe memory/notification failure was also captured before that exception. There is not yet a small standalone workload reproducer or a patched-build comparison.

Is the scan cursor supposed to wrap or reset independently of table growth? If this is the expected mechanism, is there another cleanup path that should keep reclaiming zero-reference records once the table can no longer resize?

I've been tracking recurring fseventsd memory growth and slow file-change notifications on macOS 26.7.1 (25G241), with massive Bazel build trees. There seems to be a repeatable boundary inside event-table cleanup. I may be missing another reset path, but the live observations match the disassembly closely.

What's you're role in that process? Are you one of the monitoring processes or are these system processes that are already monitoring? How fast it the Basel hierarchy growing and does it need to be actively monitored?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin. I'm debugging this as the user of the Mac. Finder and other existing applications are FSEvents clients, and the CPU/memory problems predate the diagnostic monitoring. In fact, they were the reason I started doing it.

The monitor I added samples fseventsd once a minute. A notification test briefly starts fswatch on a small private probe directory, creates a file, measures the notification delay, and stops the listener. That listener isn't watching the Bazel trees. Then it does a cursor/record inspection by reading the daemon's memory without suspending it or writing. Probe only.

I dont have a reliable files/sec or FSEvents/sec figure for the build trees. There is substantial file creation and removal during builds; the fs_usage counts I have are syscall counts, so they wouldn't establish that rate. I haven't isolated which existing subscribers need to watch those trees or whether their watches can be narrowed. That's up to them, and I dont have any direct control there. But heavy use shouldn't really break things.

What I need is for normal file-change notifications to keep working, including Finder/Desktop updates, without the sudden memory bloat and CPU load. I'm still treating the Desktop symptom separately. My concern with the cleanup code is that, once the cursor stops, zero-reference records seem to accumulate until a resize resets it. At the maximum table size, that recovery route is no longer available and fseventsd begins accumulating memory and CPU like a runaway snowball.

I’m seeing similar symptoms on macOS 15.7.9 (24G830) with fseventsd 1400.100.1, so this may also affect an older macOS release. Details here FB25224901

Before restarting the service, diagnostics showed:

• Approximately 2.9 GiB physical footprint.

• 5,212,296 live PathStringZone allocations, occupying approximately 1.6 GiB, with only about 1% fragmentation.

• Approximately 2.8 GiB of writable memory marked swapped out.

• The active notification-thread samples predominantly reached strncmp through the interned-string lookup chain.

• A short filesystem trace recorded approximately 395 MiB of page-in requests over 5.4 seconds, almost entirely on the notification thread and backed by swap. These are summed request sizes, not unique pages or a sustained-rate measurement.

Approximately six minutes after restarting only fseventsd, its footprint was 4,577 KiB, with 1,721 PathStringZone allocations. A separate short comparison trace recorded no page-in requests.

Read-only inspection of this installed version also found the mechanism described here: a 32-bit cleanup cursor at event-table offset 0x14, advancing by 512 per complete pass, with cleanup skipped above 0xfffffdff. Starting from zero, that reaches 0xfffffe00 after 8,388,607 complete passes. Startup and successful table resizing reset the cursor; the inspected resize limit is 262,144 buckets.

Important qualification: I did not capture the affected process’s live cursor value or zero-reference record counts before restarting it. This is therefore corroborating evidence for the hypothesis, not confirmation that cursor exhaustion caused my incident. The client mix also changed after restart, so this was not a controlled fixed-client comparison.

No debugger attachment, process-memory modification, binary patching or SIP changes were used. I have prepared a privacy-minimized diagnostic summary containing the relevant aggregate measurements and code findings.

Is there any additional narrowly scoped diagnostic that would help establish whether this older version is hitting the same failure mode?

But heavy use shouldn't really break things.

Agreed, there are definitely bugs here as, as FSEvents shouldn't impact the system under basically "any" load.

Having said that:

What I need is for normal file-change notifications to keep working, including Finder/Desktop updates, without the sudden memory bloat and CPU load.

If you're goal is to get things working better "now", then my recommendation would be to move the Bazel hierarchy into it's own file system- for example, by using an additional APFS volume within the same volume set or by creating a DiskImages.

My guess is that at least part of whats going on is that there are multiple clients monitoring the overall file systems and differences in those client configurations are causing FSEvents to hold on to a lot more event data than it "should" and, with luck, moving the data to a separate file system will help mitigate that.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware