Caching and tiering

There are two caches in play, one inside the other. A node cache is the block device a storage node keeps for caching: the EBS volumes striped into one RAID0 device. A volume cache is the slice of it a volume gets for itself, carved out of the node cache of whichever node the volume was placed on.

A volume's slice is one area, not two. Its read size is the whole footprint on disk, and it serves reads and writes alike; its write size is a limit inside that footprint — how much of it may be dirty and waiting to drain to object storage. So the write size never asks the disk for another byte.

The node cache is a Terraform decision, fixed when the pool is applied. The volume caches are a StorageClass decision, sized from the volume's size when the claim is created. Both have to add up, which is what the sizing sections at the end are for.

#Where the caches live

Those slices are per volume, but the disk under them is not. Every volume placed on a node carves its cache out of the one cache device that node has:

  ┌─ storage node ─────────────────────────────────────────────────────────┐
  │  ┌─ volume A · 100Gi ───────────┐    ┌─ volume B · 20Gi ────────────┐  │
  │  │ cache        10240 MiB       │    │ cache         2048 MiB       │  │
  │  │  └ dirty ≤    3072 MiB       │    │  └ dirty ≤    1024 MiB       │  │
  │  │ QoS  250 MB/s · 20000 IOPS   │    │ QoS  100 MB/s ·  5000 IOPS   │  │
  │  └──────────────┬───────────────┘    └──────────────┬───────────────┘  │
  │                 └─────────────────┬─────────────────┘                  │
  │                                   ▼                                    │
  │  ┌─ RAID0 cache device ─────────────────────────────────────────────┐  │
  │  │  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌────────────┐  │  │
  │  │  │   EBS 1    │  │   EBS 2    │  │   EBS 3    │  │   EBS N    │  │  │
  │  │  └────────────┘  └────────────┘  └────────────┘  └────────────┘  │  │
  │  │  one filesystem, one size budget, and one IOPS + throughput      │  │
  │  │  budget, shared by every volume placed on this node              │  │
  │  └──────────────────────────────────────────────────────────────────┘  │
  └────────────────────────────────────────────────────────────────────────┘
                                      │ misses and flushes · 1 MiB blocks
                                      ▼
                                ┌───────────┐
                                │    S3     │
                                └───────────┘

A node's cache disks are therefore a budget shared by everything placed on it — and it is two budgets, not one. Size is the obvious one. The other is performance: the RAID0 device has a finite IOPS and throughput, the sum of what its EBS volumes provide, and every volume on the node draws its reads, its writes and its flushes from that same pool. A volume's QoS caps (qosRWIOPS, qosRMBPerSec, qosWMBPerSec, qosRWMBPerSec) are what bound one volume's share of it; left unset, a volume has no cap and one busy workload can take the device.

So tune both. Cache sizes decide how many volumes fit; QoS caps decide whether the volumes that fit can all be served at once. Ideally the sum of the volumes' caps stays inside the node's RAID0 IOPS and throughput, so a full node is still a node where every volume gets the performance its StorageClass promises.

#Reads

A read whose block is in the read cache is served from local disk. A read that misses fetches the 1 MiB block containing it from object storage, so a cold volume — freshly created, freshly restored, or one whose working set has been evicted — pays an object-storage round trip until its working set is resident again.

Sequential access is cheap in this model, because a 1 MiB block satisfies a lot of consecutive reads. What a miss costs, and what a cold volume costs before its working set is resident, are in Tradeoffs.

#Writes

A write is acknowledged once it is in the write cache; the flush to object storage happens in the background, coalescing dirty ranges into whole 1 MiB blocks so that a page of small random writes does not become a page of uploads.

Two consequences worth being explicit about:

• Recently written blocks are on local disk before they are in the bucket.
• A volume that is written faster than its cache can drain will eventually be limited by the drain rate rather than by the cache — see Tradeoffs. QoS caps are the tool for keeping one volume from taking the whole pool's drain capacity.

#Idle volumes

A volume that no pod has used for controller.idleVolumeMin minutes (10 by default) is stopped on the backend by the controller's reconciler. The volume itself is untouched; the next attach starts it again, and it warms up from object storage the same way a new one does. Its cache slices stay reserved while it is idle, so this returns CPU and memory to the node, not capacity to the pool — see Idle volumes.

#How the caches are sized

Sizing a pool right now? Sizing the pool runs the arithmetic in this section for you — the cache disks a node needs, the two per-disk pool sizes, and the StorageClass ratios and ceilings that go with them.

#The node cache

Set per pool, in Terraform. r_cache_size_in_mib and rw_cache_size_in_mib are per disk, not per node.

#A volume's caches

Set per StorageClass, and applied when the claim is created. These are the per-volume counterparts of the pool variables above — six for size, four for performance — and the defaults are the chart's:

The two cache sizes are computed once, when the volume is created, from the ratio and the two bounds:

cache = clamp(ratio * volume size in MiB, floor, ceiling)

So the ratio governs mid-sized volumes, the floor governs small ones, and the ceiling governs everything above the crossover at ceiling / ratio. Both sizes are computed this way, and the write one has to come out below the read one — it is a share of the same area.

The four QoS parameters are absolute numbers rather than ratios, and unset means no cap at all rather than a zero. They are the volume's share of the node's RAID0 performance, which is the last thing the next section adds up.

Volumes with different cache or QoS profiles mean different StorageClasses. That is the intended way to run, say, a database volume and an archive volume against the same pool.

#Making the two add up

First, the node cache has to fit its disks. Reserve about 7% for metadata and the filesystem; what is left is what the read size may claim, because that is the space the caches occupy:

available = disk_GiB * 1024 * 0.93        # per disk
r_cache_size_in_mib <= available
rw_cache_size_in_mib <= r_cache_size_in_mib

The write size is bounded by the read size rather than by the disk — it is how much of that space may be dirty at once, so setting it higher buys nothing and setting it to zero would leave the cache no room to absorb a write.

Then, the volumes have to fit the node cache. Multiply the per-disk sizes by the disks to get a node's budget, and divide by what one volume reserves:

read budget  = r_cache_size_in_mib  * nvme_node_disks_count
write budget = rw_cache_size_in_mib * nvme_node_disks_count

volumes per node = min( max_volumes_count - 1,
                        read budget  / volume read cache,
                        write budget / volume write cache )

Both budgets are still checked separately — the scheduler tracks the read and the write reservations of every volume on a node — so a write budget set too low stops placement just as a read budget does, even though it costs no disk. Which of the three binds is worth knowing rather than guessing.

Last, the volumes' QoS has to fit the disks' performance. Cache size decides how many volumes a node accepts; it says nothing about whether they can all be served at once. Add up the caps the StorageClasses hand out and compare them against what the RAID0 device actually delivers — the iops and throughput on ebs_volumes, times the disk count:

node IOPS       = ebs_volumes.iops       * nvme_node_disks_count
node throughput = ebs_volumes.throughput * nvme_node_disks_count

volumes per node * qos_rw_ios_per_sec    <= node IOPS
volumes per node * qos_rw_mbytes_per_sec <= node throughput

Widen the stripe before you provision it. gp3 grants every disk 3000 IOPS and 125 MB/s free and charges for capacity alone, so those two figures are the only ones on the invoice that come per disk rather than per gigabyte: the same cache spread over ten disks costs what it costs on three and delivers 30,000 IOPS and 1,250 MB/s instead of 9,000 and 375. Build the stripe at ten disks whatever the volumes ask for — it is the cheapest performance in the pool, and a node that never touches its headroom has paid nothing for it. Only a node that needs more than 30,000 IOPS or 1,250 MB/s has any reason to provision a disk above the baseline. Ten is where it stops: past that it is more attachments and more to lose for the same capacity, and the instance's own EBS bandwidth is usually the ceiling long before the disks are.

The scheduler checks the volume count and the cache budgets before it places a volume, so a pool that is out of cache budget stops accepting volumes even with the buckets empty — see Volume scheduling.

QoS is not checked against disk performance, and nothing stops you from handing out caps that add up to more than the stripe delivers. That arithmetic is yours precisely so that overprovisioning stays available as a choice: more volumes per node, at a lower cost each, paid for in peak performance on the occasions they are all busy at once. Whether that is a good trade is a property of the workload — it usually is for volumes that are busy in turns, and it is not for volumes that peak together.

#Calculation example

A database pool: large volumes, a hot fifth of each one that has to stay on local disk, and performance promised per volume rather than in aggregate.

1. How many nodes, and which instance. Three nodes before any arithmetic — that is the floor for replication and quorum whatever the capacity works out to, and the reasoning is in High availability. Fifteen volumes across three nodes is five each, which is the number the rest of this depends on:

per node = 5 * 6000 =  30000 IOPS
           5 *  200 =   1000 MB/s

Five volumes per node is also what picks the instance. Each volume takes roughly half a core and 2 GiB at 2Ti, and together they need the EBS bandwidth of their caps plus half again of it on the network for the S3 side:

cores   = 2 (SPDK) + 5 * 0.5 + 1  =  5.5   → 4 vCPU is not enough
memory  = 4 + 5 * 2 + 4           =   18   GiB
EBS     = 5 * 200                 = 1000   MB/s
network = 5 * 200 * 1.5           = 1500   MB/s ≈ 12 Gbit

An m8gb.xlarge clears none of the four, at 4 vCPU, 16 GiB, 781 MB/s to EBS and 8.3 Gbit. So m8gb.2xlarge — 8 vCPU, 32 GiB, 1562 MB/s and 16.7 Gbit — with room for the figures to move once they are measured; see Node count and instance type.

2. The stripe that delivers the performance. Ten disks, on the baseline. gp3 is provisioned per disk and RAID0 adds it up, so ten disks carrying nothing but their free 3000 IOPS and 125 MB/s cover the whole requirement:

node IOPS       = 10 * 3000 = 30000       >= 30000 needed
node throughput = 10 *  125 =  1250 MB/s  >=  1000 needed

Six disks at 6000 IOPS and 250 MB/s would serve the same five volumes — and cost $120 a month a node to do it, $20 a disk for the 3000 IOPS and 125 MB/s each one is provisioned above its baseline. The ten-disk stripe holds the same bytes for the same capacity bill and asks for no performance at all. Headroom above the requirement is the one thing it does not have, since 30,000 lands exactly on the demand; buying it back is cheap — iops = 3200 is 32,000 across the stripe for $10 a node — but it is a choice rather than the default.

3. What one volume needs from the cache. The working-set target is a ratio, not a fixed size, so it goes in as one. The read figure is the whole slice; the write figure is how much of it may be dirty at once, which only has to absorb a burst until it drains:

read  = 0.2  * 2097152 = 419430 MiB      # 410 GiB on disk, the resident fifth
write = 0.02 * 2097152 =  41943 MiB      #  41 GiB of it dirty at once

So 410 GiB per volume, a fifth of 2Ti — the write figure adds nothing to that.

4. What five of them need from a node. Five volumes at a fifth each is one whole volume's worth of cache per node. Divide by the ten disks, because the pool variables are per disk, and round up so each budget clears its demand:

read  = 5 * 419430 = 2097152 MiB   →  r_cache_size_in_mib  = 210000 * 10 = 2100000  ✓
write = 5 *  41943 =  209715 MiB   →  rw_cache_size_in_mib =  22000 * 10 =  220000  ✓

The read line is what will size the disks. The write line is a reservation the scheduler checks and nothing more.

5. How big the disks have to be. Only r_cache_size_in_mib occupies space, so a disk has to be big enough that 93% of it clears 210000 — 221 GiB, rounded to 225:

available   = 225 * 1024 * 0.93 = 214272 MiB per disk   >= 210000  ✓
node usable = 214272 * 10       = 2142720 MiB           98% claimed

The disk count does not move: ten disks were chosen in step 2 because ten is what the free allowance is granted on, and residency does not change what the volumes were promised. Only size grows.

6. The pool, in Terraform.

nodes_count           = 3
nodes_instance_type   = "m8gb.2xlarge"  # 8 vCPU, 32 GiB

raid_level            = 0
nvme_node_disks_count = 10
max_volumes_count     = 6               # strict compare, so five land

r_cache_size_in_mib   = 210000          # per disk
rw_cache_size_in_mib  =  22000          # per disk

ebs_volumes = [{
  size       = 225
  type       = "gp3"
  iops       = 3000                     # baseline; 30000 across the stripe
  throughput = 125                      # baseline;  1250 MB/s across the stripe
  count      = 10
}]

7. The volumes, in the StorageClass. The ratios carry the working-set intent, so a 1Ti volume on this class gets its own fifth rather than the same fixed slab. The ceilings sit just above what a 2Ti volume asks for — deliberately, so a larger claim is clamped instead of overrunning the node budget. The requirement itself becomes the two QoS caps:

parameters:
  labels: "storage=mgx,profile=db"
  ratio_cache_r_cache_size: "0.2"       # the working-set target
  ratio_cache_rw_cache_size: "0.02"
  max_cache_r_cache_size: "420000"      # default 20480; binds above 2050 GiB
  max_cache_rw_cache_size: "42000"      # default 3072
  qos_rw_ios_per_sec: "6000"
  qos_rw_mbytes_per_sec: "200"

8. Check it end to end. Every guard, on the fifth volume of a node — the one that has to pass:

per volume     read  clamp(0.2  * 2097152, 1024, 420000) = 419430 MiB on disk
               write clamp(0.02 * 2097152, 1024,  42000) =  41943 MiB dirty

disk fit       210000          =  210000 <= 214272   ✓   225 GiB, less 7%
volume count   4 + 1           =       5 <       6   ✓   a sixth is rejected
read cache     5 * 419430      = 2097152 < 2100000   ✓   210000 * 10 disks
write cache    5 *  41943      =  209715 <  220000   ✓    22000 * 10 disks
node IOPS      5 *   6000      =   30000 <=  30000   ✓      3000 * 10 disks, free
node MB/s      5 *    200      =    1000 <=   1250   ✓       125 * 10 disks, free
node cores     2 + 2.5 + 1     =     5.5 <=      8   ✓
node memory    4 + 10 + 4      =      18 <=     32   ✓   GiB

9. What it adds up to. Fifteen volumes, each with the 6000 IOPS and 200 MB/s it was asked to have and 410 GiB of cache behind it — enough that reads inside the working set stop leaving the node, so the IOPS cap is what the database feels rather than a cap in front of an S3 round trip. The pool holds 30 TiB of logical capacity on 6 TiB of cache, 5:1, over 6750 GiB of EBS — and every IOP and megabyte a second of it comes free with those gigabytes.

That last number is the shape of the trade: residency is bought in gigabytes, and the performance comes with them as long as the stripe stays ten disks wide. A class that kept the same performance on a tenth of the volume resident would need half the EBS and the same 30,000 IOPS, because the baseline is granted per disk whatever the disk's size — ten 111 GiB disks carry it as readily as ten 225 GiB ones.

What it deliberately does not have. The three nodes are exactly full, so a node lost or drained has nowhere to put its five volumes — a fourth node is the price of surviving one. And nothing is oversubscribed: the stripe can serve all five volumes at their caps at once — with nothing to spare on IOPS, since ten baseline disks land exactly on the 30,000 the five were promised. Raising the caps above node IOPS / 5 and node throughput / 5 is how you trade that guarantee for more volumes per node, and it is a reasonable trade when volumes are busy in turns rather than together.