Post

Replies

Boosts

Views

Activity

Reply to smbfs silently zero-fills already-written data after cached file size regresses on reopen
Thank you for looking into this case, Kevin. Indeed, we have filed a report (with pcap, DTrace output, byte-level diffs, etc.) - it is FB24552903 (https://feedbackassistant.apple.com/feedback/24552903). If it would help you narrow the case down, please let me know, and I can run variants of the workload. Thank you in advance! Regarding your comment on the workload - it is a fair point. Let me give some context on why it looks this way. We have an SMB enterprise solution, and our customers run it under just about every application imaginable - video editing, audio workstations, render pipelines, build systems. So we don't get to choose the I/O patterns; we have to survive all of them. We therefore run long, multi-node simulations that exercise the whole chain and independently verify every byte written, because data integrity is the one property we can't compromise on. The pattern I posted is a distilled reproducer from those simulations: deliberately adversarial, because its job is to compress a months-scale field risk into hours. On the reordering example: I agree that's a real interleaving, and we see that transient zero-fill constantly - but it's self-healing. In your third ordering, thread 2's write(2) has already returned successfully, so its bytes sit in the UBC as dirty pages and flush afterward; the final file is whole. That case is not harmful in any way, and it isn't what I'm reporting. I would say there are three things that separate the observed failure from that race: Nothing was in flight against the zeroed range.The writer runs 8 threads per batch over disjoint 8 KiB ranges and joins all of them before the next batch starts. The zeroed range [147456:196608) belonged to a completed batch: every write(2) covering it had returned, the client had flushed it (smbfs_do_strategy writes at offsets 147456/163840/180224), and the server had acknowledged a strictly monotonic EOF through 196608. The zeros arrived ~50 ms later, on the same ordered SMB2 session, from the next batch. There was no pending write left to reorder with. The discriminator is the size regression, which no legal interleaving produces. 46 ms after the last correct flush, the kernel logged smbfs_update_size: UBC_PUSHDIRTY, UBC_INVALIDATE ... due to file size changed, and the next vnop_write entered with old eof = 147456 - backwards from 196608. That eviction is what destroys the already-flushed pages (so nothing can re-cover the range), and the regressed n_size is what aims the zero-fill at committed data. 147456 is a value the client itself held 84 ms earlier; it appears in none of the server's size-bearing replies - CREATE, CLOSE, QUERY_INFO, or QUERY_DIRECTORY - all checked in the capture. The failure discriminates in a way an ordering race can't. Each file is written twice from the same buffer by the same code - NAME, then NAME.copy. Across every instance we've captured, NAME is intact and only the second carries the hole, and the hole is always exactly [regressed n_size : next write-aligned ...]. A reorder-driven zero-fill would have no reason to respect that structure. So while the workload is intentionally unkind, a server-acknowledged write can be silently zeroed afterward because the client's cached EOF moved backwards.
Topic: App & System Services SubTopic: Core OS Tags:
Sep ’26
Reply to smbfs silently zero-fills already-written data after cached file size regresses on reopen
Good suggestion, tested it. I added os.fsync(fd) between the last write() and close() of every chunk, so each 8 KiB chunk's handle was fsync'd before closing. The corruption still reproduces. It hit at 37 minutes in (previous runs ranged 10 min to 3 h, so well within normal variance). Same signature: exactly one 8 KiB chunk of zeros, [204800:212992), at the correct total file length. The fsync had no measurable effect on the mechanism, and the dtrace capture from this run shows why. The zeros don't come from unflushed data. The correct bytes were already transmitted and acknowledged by the server (visible in the packet capture); the zeros come from a later write consulting a stale cached file size. In this run the trace actually caught the trigger directly: the file's vnode was reclaimed and re-instantiated three times within one second, mid-write. On one of the freshly created vnodes (smbfs_nget path), smbfs_update_size applied a stale EOF of 196608 eight times in a row while the file had already grown past 212992; the next write then zero-filled the "gap" [204800:212992) via IO_HEADZEROFILL and flushed the zeros over data the server already had. So fsync guarantees the data reaches the server, but this bug doesn't lose unflushed data. It actively overwrites already-acknowledged data, driven by the attribute-cache / vnode-lifetime race.
Topic: App & System Services SubTopic: Core OS Tags:
Sep ’26
Reply to smbfs silently zero-fills already-written data after cached file size regresses on reopen
Thank you for looking into this case, Kevin. Indeed, we have filed a report (with pcap, DTrace output, byte-level diffs, etc.) - it is FB24552903 (https://feedbackassistant.apple.com/feedback/24552903). If it would help you narrow the case down, please let me know, and I can run variants of the workload. Thank you in advance! Regarding your comment on the workload - it is a fair point. Let me give some context on why it looks this way. We have an SMB enterprise solution, and our customers run it under just about every application imaginable - video editing, audio workstations, render pipelines, build systems. So we don't get to choose the I/O patterns; we have to survive all of them. We therefore run long, multi-node simulations that exercise the whole chain and independently verify every byte written, because data integrity is the one property we can't compromise on. The pattern I posted is a distilled reproducer from those simulations: deliberately adversarial, because its job is to compress a months-scale field risk into hours. On the reordering example: I agree that's a real interleaving, and we see that transient zero-fill constantly - but it's self-healing. In your third ordering, thread 2's write(2) has already returned successfully, so its bytes sit in the UBC as dirty pages and flush afterward; the final file is whole. That case is not harmful in any way, and it isn't what I'm reporting. I would say there are three things that separate the observed failure from that race: Nothing was in flight against the zeroed range.The writer runs 8 threads per batch over disjoint 8 KiB ranges and joins all of them before the next batch starts. The zeroed range [147456:196608) belonged to a completed batch: every write(2) covering it had returned, the client had flushed it (smbfs_do_strategy writes at offsets 147456/163840/180224), and the server had acknowledged a strictly monotonic EOF through 196608. The zeros arrived ~50 ms later, on the same ordered SMB2 session, from the next batch. There was no pending write left to reorder with. The discriminator is the size regression, which no legal interleaving produces. 46 ms after the last correct flush, the kernel logged smbfs_update_size: UBC_PUSHDIRTY, UBC_INVALIDATE ... due to file size changed, and the next vnop_write entered with old eof = 147456 - backwards from 196608. That eviction is what destroys the already-flushed pages (so nothing can re-cover the range), and the regressed n_size is what aims the zero-fill at committed data. 147456 is a value the client itself held 84 ms earlier; it appears in none of the server's size-bearing replies - CREATE, CLOSE, QUERY_INFO, or QUERY_DIRECTORY - all checked in the capture. The failure discriminates in a way an ordering race can't. Each file is written twice from the same buffer by the same code - NAME, then NAME.copy. Across every instance we've captured, NAME is intact and only the second carries the hole, and the hole is always exactly [regressed n_size : next write-aligned ...]. A reorder-driven zero-fill would have no reason to respect that structure. So while the workload is intentionally unkind, a server-acknowledged write can be silently zeroed afterward because the client's cached EOF moved backwards.
Topic: App & System Services SubTopic: Core OS Tags:
Replies
Boosts
Views
Activity
Sep ’26
Reply to smbfs silently zero-fills already-written data after cached file size regresses on reopen
Good suggestion, tested it. I added os.fsync(fd) between the last write() and close() of every chunk, so each 8 KiB chunk's handle was fsync'd before closing. The corruption still reproduces. It hit at 37 minutes in (previous runs ranged 10 min to 3 h, so well within normal variance). Same signature: exactly one 8 KiB chunk of zeros, [204800:212992), at the correct total file length. The fsync had no measurable effect on the mechanism, and the dtrace capture from this run shows why. The zeros don't come from unflushed data. The correct bytes were already transmitted and acknowledged by the server (visible in the packet capture); the zeros come from a later write consulting a stale cached file size. In this run the trace actually caught the trigger directly: the file's vnode was reclaimed and re-instantiated three times within one second, mid-write. On one of the freshly created vnodes (smbfs_nget path), smbfs_update_size applied a stale EOF of 196608 eight times in a row while the file had already grown past 212992; the next write then zero-filled the "gap" [204800:212992) via IO_HEADZEROFILL and flushed the zeros over data the server already had. So fsync guarantees the data reaches the server, but this bug doesn't lose unflushed data. It actively overwrites already-acknowledged data, driven by the attribute-cache / vnode-lifetime race.
Topic: App & System Services SubTopic: Core OS Tags:
Replies
Boosts
Views
Activity
Sep ’26