The write path: what a write-back cache buys, and what it costs

A filesystem expects a disk. It issues small writes, often to addresses it picked at random, and it expects each one acknowledged before it moves on. S3 offers none of that: an object is written whole, over HTTPS, and there is no such thing as overwriting the middle of one.

Everything interesting about running a block device on object storage happens in the gap between those two sentences. The gap is closed by a write-back cache on local disk, and the shape of that cache decides what the volume feels like.

#A write is acknowledged on local disk

Each volume gets a write cache on the storage node's local disks. A write lands there, is acknowledged, and the pod moves on. The flush to the bucket happens afterwards, in the background, on the storage node's own schedule rather than the workload's.

   pod ──▶ NVMe target ──▶ write cache          acknowledged here
                                │
                                │ background flush
                                ▼
                       S3 (1 MiB objects)

The flush is where the object-storage semantics get paid for. Dirty ranges are coalesced into whole 1 MiB blocks, so a page of small random writes leaves as one object rather than as a few hundred requests. That is not an optimisation bolted on afterwards; it is the reason the design works at all. An object is replaced whole, so the unit that leaves the node has to be the whole block, and a cache that flushed every write as it arrived would turn a database's write pattern into a request-per-write bill.

The same 1 MiB unit is what makes reads cheap in the other direction: one fetched block satisfies a lot of consecutive reads.

#The cache is sized from the volume, at creation

Both caches are a ratio of the volume's size, clamped between a floor and a ceiling, and all three numbers come from the StorageClass rather than from anything discovered at runtime. On the default class:

So a 100Gi volume works out as:

read  cache = clamp(0.1  * 102400, 1024, 20480) = 10240 MiB
write cache = clamp(0.05 * 102400, 1024,  3072) =  3072 MiB

Because of the ceilings, the write cache stops growing at a 60Gi volume and the read cache at 200Gi. Past that, a bigger volume does not get a bigger cache unless you raise the ceilings, and a volume that wants a different profile wants a different StorageClass — which is the intended way to run a database volume and an archive volume against the same pool. The full arithmetic is on the caching page.

#Two things this costs you

Acknowledging a write before it is durable in the bucket buys the latency. It also buys two consequences that are worth saying out loud rather than discovering.

Recently written blocks are on a local disk before they are in S3.

A volume written faster than its cache drains is limited by the drain, not by the cache. The tool for keeping one volume from spending the whole pool's drain capacity is the per-volume QoS caps.

#Reads are the mirror image

A read whose block is resident is served from local disk. A read that misses fetches the 1 MiB block containing it from the bucket first, so a cold volume — freshly created, freshly restored, or one whose working set has been evicted — pays object-storage round trips until it is warm again.

That puts the expensive case in plain view: random reads across a working set larger than the read cache. Sequential access is cheap, a warm working set is cheap, and everything else is a question of whether the cache is big enough. It is the same question the sizing table above answers, which is why cache sizing is the first thing to tune and the last thing to guess at.

#Every technology has tradeoffs

A storage system that presents object storage as a disk can either pretend the bucket is not there or tell you exactly where it is. The second is more useful: a cache hit behaves like the local disk it is, a miss behaves like the S3 GET it is, and both are things you can reason about, size for and cap.

The tradeoffs section of the docs is the short version of that, and the architecture page is the long one.