System-wide deadlock in removexattr from revisiond / APFS

Question

Created 22h

Replies 1

Boosts 0

Participants 2

System-wide deadlock in removexattr from revisiond / APFS

We're experiencing a deadlock on certains systems when our software is installed, which is causing side effects in our process (and likely others) such as blocked queues and increased memory usage.

According to the spindump, revisiond appears to be holding an exclusive lock within the kernel.

Process:          revisiond [426]
UUID:             5E9B9E04-984B-31AD-A4FF-A1A90B7D53A1
Path:             /System/Library/PrivateFrameworks/GenerationalStorage.framework/Versions/A/Support/revisiond
Codesigning ID:   com.apple.revisiond
Shared Cache:     25AE5A2A-FE2A-3998-8D4E-F3C5C6E6CEB6 slid base address 0x189834000, slide 0x9834000 (System Primary)
Architecture:     arm64e
Parent:           launchd [1]
UID:              0
Sudden Term:      Tracked
Memory Limit:     50MB
Jetsam Priority:  40
Footprint:        6225 KB
Time Since Fork:  1740319s
Num samples:      940 (1-940)
Num threads:      5
Note:             1 idle work queue thread omitted

[...]

  Thread 0xc0616d    940 samples (1-940)    priority 46 (base 4)    last ran 241692.754s ago
  940  start_wqthread + 8 (libsystem_pthread.dylib + 7068) [0x189d0ab9c]
    940  _pthread_wqthread + 292 (libsystem_pthread.dylib + 11852) [0x189d0be4c]
      940  _dispatch_workloop_worker_thread + 692 (libdispatch.dylib + 85356) [0x189b65d6c]
        940  _dispatch_root_queue_drain_deferred_wlh + 292 (libdispatch.dylib + 87156) [0x189b66474]
          940  _dispatch_lane_invoke + 440 (libdispatch.dylib + 45048) [0x189b5bff8]
            940  _dispatch_lane_serial_drain + 944 (libdispatch.dylib + 42420) [0x189b5b5b4]
              940  _dispatch_client_callout + 16 (libdispatch.dylib + 113364) [0x189b6cad4]
                940  _dispatch_call_block_and_release + 32 (libdispatch.dylib + 7004) [0x189b52b5c]
                  940  ??? (revisiond + 168768) [0x10494d340]
                    940  ??? (revisiond + 165940) [0x10494c834]
                      940  ??? (revisiond + 40264) [0x10492dd48]
                        940  ??? (revisiond + 56680) [0x104931d68]
                          940  <patched truncated backtrace>
                            940  removexattr + 8 (libsystem_kernel.dylib + 23768) [0x189cd1cd8]
                             *940  ??? (kernel.release.t6000 + 15240) [0xfffffe000886fb88]
                               *940  ??? (kernel.release.t6000 + 1886348) [0xfffffe0008a3888c]
                                 *940  ??? (kernel.release.t6000 + 7730436) [0xfffffe0008fcb504]
                                   *940  ??? (kernel.release.t6000 + 2759592) [0xfffffe0008b0dba8]
                                     *940  ??? (kernel.release.t6000 + 2808244) [0xfffffe0008b199b4]
                                       *940  apfs_vnop_removexattr + 1044 (apfs + 474512) [0xfffffe000be8d4d0]
                                         *940  decmpfs_cnode_set_vnode_state + 80 (kernel.release.t6000 + 2945816) [0xfffffe0008b3b318]
                                           *940  IORWLockWrite + 184 (kernel.release.t6000 + 496184) [0xfffffe00088e5238]
                                             *940  ??? (kernel.release.t6000 + 494624) [0xfffffe00088e4c20]
                                               *940  ??? (kernel.release.t6000 + 619452) [0xfffffe00089033bc]
                                                 *940  ??? (kernel.release.t6000 + 624472) [0xfffffe0008904758]

The bulk of the other processes are waiting for that lock.

(suspended, blocked by krwlock for reading owned by revisiond [426] thread 0xc0616d)

(blocked by krwlock for writing owned by revisiond [426] thread 0xc0616d)

Around the time of the event, these messages were logged by revision:

2026-03-06 18:49:37.781673-0500 0x16b7     Error       0x7f92f364           426    14   revisiond: [com.apple.revisiond:default] [ERROR] CSCopyChunkIDsForToken failed for 41639
2026-03-06 18:49:37.781716-0500 0x16b7     Error       0x7f92f365           426    14   revisiond: [com.apple.revisiond:default] [ERROR] updateEntry for new entry <private> failed
2026-03-06 18:49:37.781738-0500 0x16b7     Error       0x7f92f366           426    14   revisiond: [com.apple.revisiond:default] [ERROR] no entry for '<private>'
2026-03-06 18:49:37.781754-0500 0x16b7     Error       0x7f92f367           426    14   revisiond: [com.apple.revisiond:default] [ERROR] failed assembleInfoForOffset for fsid 16777234 fileid 359684022 offset 0 size 14334 (path <private>)

Our agent uses Endpoint Security Framework to monitor events, and provide anti-tamper functionality for installed components and processes. While several EndpointSecurity calls appear in the spindump stack traces, we don't have any evidence that any calls from revisiond were blocked.

What we'd really like to to understand what that lock is (appears to be decompressing an object on an APFS volume), what revisiond and APFS are doing with it, and what might cause it to deadlock.

Of note, one of our processes is also waiting on that lock, one thread for reading and the other for writing.

This issue affects machines running several macOS versions (15.x, 26.x). The machine in the examples is running macOS 26.3 (25D125)

Boost

Answer 1

DTS Engineer OP

Apple

35m

We're experiencing a deadlock on certains systems when our software is installed, which is causing side effects in our process (and likely others) such as blocked queues and increased memory usage.

According to the spindump, revisiond appears to be holding an exclusive lock within the kernel.

Can you file a bug on this and upload the full spindump there? That's the easiest way to transfer large data files and I may eventually want a sysdiagnose.

Having said that...

Our agent uses Endpoint Security Framework to monitor events, and provide anti-tamper functionality for installed components and processes. While several EndpointSecurity calls appear in the spindump stack traces, we don't have any evidence that any calls from revisiond were blocked.

...this is almost CERTAINLY caused by your ES client.

What we'd really like to to understand what that lock is (appears to be decompressing an object on an APFS volume), what revisiond and APFS are doing with it, and what might cause it to deadlock.

You can actually find the code for decmpfs_cnode_set_vnode_state here, however, I think you're asking the wrong question. The question here isn't what revisiond is doing, it's who revisiond is waiting on. Some other thread locked that lock and is now stuck, which is the real issue you need to sort out.

Case in point, you're getting closer:

Of note, one of our processes is also waiting on that lock, one thread for reading and the other for writing.

However, the problem thread ISN'T going to be waiting on that lock- that thread is going to be stuck "somewhere else", unable to complete it's work and release the lock.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

0