TECH Signal 493
Every fast write moves work somewhere else
↗
Fast write acknowledgments shift durability work to other layers, trading latency for risk of data loss.
Engineers must know exactly what guarantees are in place when a write is reported successful, because the remaining risk determines what data can be lost and what recovery steps are required. Understanding where the work moves helps in choosing appropriate storage configurations and in designing correct crash-recovery logic.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
A write that returns after copying to memory is fast but can be lost on any crash.
Waiting for a local SSD sync survives process or kernel crashes but not loss of the drive or host.
Offloading durability to remote storage or a quorum of replicas adds network latency but tolerates more failures.
THE READ
What the cluster adds up to.
A storage engine must decide when it is safe to tell a client that a write succeeded. The fastest option is to acknowledge after the bytes have been copied into memory, which moves the durability work to later steps. This approach minimizes latency but leaves the write vulnerable to any crash that loses memory. The trade-off is therefore between immediate response time and the risk of data loss.
When the engine waits for an fdatasync on a host-local NVMe SSD, the acknowledgment point shifts to after the device has flushed its cache. This protects the write against process or kernel crashes, but a failure of the SSD or the whole host can still erase the data. The latency of this path depends on how long the device takes to complete the flush, which can vary widely. Engineers must weigh the added durability against the possible increase in response time.
Alternatively, the engine can treat the NVMe interface as a gateway to a remote storage service or wait for a quorum of replica nodes to persist the write. In this case the acknowledgment occurs after the network round-trip and the remote service’s own durability guarantees, which adds latency but tolerates loss of the local drive or even the entire host. The same system call name (fdatasync) can hide very different failure coverage depending on where the storage actually resides. Choosing this path moves work from the local host to the network or to other nodes.
Newer designs often combine an object store for immutable sorted files with a host-local NVMe write-ahead log and an LSM layout. After a successful object PUT the storage service owns the durability of those bytes, so the database only needs to track the current version, serve reads quickly, purge old versions, and recover from crashes. The write-ahead log can make the client PUT fast, but the system must decide whether losing that log is acceptable or whether another copy must exist before reporting success. Consequently, a fast acknowledgment may leave unfinished cleanup work such as merging LSM levels or deleting obsolete files that must be handled later.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER