Tradeoffs

Every technology has tradeoffs. This document outlines key design choices to help inform your evaluation.

The design is one deliberate bet: object storage holds the data, local disk holds what is being worked on. Everything below follows from that.

#A read that misses is slow

Object storage as the durable tier is what makes the economics work, and it is also where the costs are.

A read that hits the cache never leaves the node; a read that misses fetches the 1 MiB block containing it from S3, so a cold volume is slower than a warm one until its working set is resident.

   read hit    pod ──▶ cache                        ~2-10 ms
   read miss   pod ──▶ cache ──▶ S3 (1 MiB GET)   ~50-150 ms

Sequential access barely notices, because one 1 MiB block answers a lot of consecutive reads. Random reads across a working set larger than the cache are the case that hurts, and the tail is what the application feels: not a slower average, but occasional reads that take orders of magnitude longer than the rest.

Cache sizing is therefore the tuning knob that matters, and it is a capacity decision made per StorageClass — see Caching and tiering. Sizing the cache to the working set is the difference between a volume that behaves like local disk and one that behaves like object storage; a volume whose working set does not fit will keep reaching for the bucket.

#Small volumes and small pools do not save money

The economics only work at size, and it is worth being blunt about where the line is.

A pool has a floor that exists whether it holds 2 TiB or 200 TiB: three storage nodes, their EBS cache, and the metadata store on each. Above that floor the marginal cost of data is S3's, which is where the advantage comes from. Below it, the floor is most of the bill.

• Volumes under about 2T spend more on their share of the pool than they save on capacity.
• Total volumes capacity under about 20 TiB is usually cheaper to run as provisioned block storage.

The advantage grows in both directions — larger volumes, and more capacity in the pool — because the same fixed cost is spread over more data. A pool of large volumes is the case this is built for; a handful of small ones is not. The cost calculator prices a specific workload against EBS gp3 if you want the crossover for yours.