Splitting EBS gp3 into tiers for cost savings: fio tests on 2 TB and 8 TB volumes

backedblock team

Object storage is quietly becoming the default home for data, because it is far cheaper than disks. Block storage charges for what you provision, not for what you use. Provision a 2 TB gp3 volume and you pay $0.08 per GB-month for all 2 TB from day one, even if half of it is empty and most of the rest is never read. S3 Standard charges $0.023 per GB-month, and only for the bytes written.

The catch is that most software can't use S3 directly. Databases, search engines, message queues and ordinary applications that work with files expect a disk. Moving them to object storage means rewriting the storage layer or switching to a different engine.

backedblock.io unlocks the cheaper tier at the block level instead. The workload gets an ordinary block device as a Kubernetes volume, hot data stays in a cache on EBS and every block ends up in S3, so the application keeps working unchanged. This post measures what that costs in performance against gp3.

Benchmark setup: the same fio tests on a tiered volume (EBS cache + S3) vs an EBS gp3 volume.

┌─ AWS EKS ──────────────────────────────────────────────────────────────┐
│                                                                        │
│  ┌──────────────────────────────┐    ┌──────────────────────────────┐  │
│  │           pod: fio           │    │           pod: fio           │  │
│  │            /data             │    │            /data             │  │
│  └───────────────┬──────────────┘    └───────────────┬──────────────┘  │
└──────────────────┼───────────────────────────────────┼─────────────────┘
                   │                                   │
                   ▼                                   ▼
   ┌─ backedblock.io ─────────────┐    ┌─ EBS gp3 ────────────────────┐
   │ ┌───────────┐  ┌───────────┐ │    │ ┌──────────────────────────┐ │
   │ │ hot       │  │ cold      │ │ VS │ │ hot + cold               │ │
   │ │ EBS gp3   │  │ S3        │ │    │ │ EBS gp3                  │ │
   │ │ cache     │  │ all blocks│ │    │ │ all blocks               │ │
   │ └───────────┘  └───────────┘ │    │ └──────────────────────────┘ │
   └──────────────────────────────┘    └──────────────────────────────┘

Summary

  1. Tiering can cut storage costs by 13–47% (46–59% with snapshots). For example, 8 TB volumes saved about 47% against gp3 for the volumes alone, and about 59% with weekly snapshots kept for six months (one full and three incrementals a month). Reserved Instances or a Savings Plan for the cache nodes can push the savings further.
  2. A cache sized to the workload performs like gp3. While the working set was in the cache, backedblock.io held every limit and kept a tighter tail under load.
  3. Cache misses and cold starts have to be measured and planned for. A read that misses costs an S3 round trip of about 120 ms, and a volume with an empty cache (like one just restored) is slow until its hot data has been read again.
  4. Small volumes and/or large working sets are cheaper on gp3. Volumes under 2 TB are mostly cheaper to run on gp3, and a workload whose working set doesn't fit in half of the volume size needs a cache so large that the savings disappear.

The setup

Benchmark cluster

ClusterEKS 1.34
Test nodesnode group of m5.2xlarge (8 vCPU, 32 GiB), Amazon Linux 2023, kernel 6.12, one node per test pod
Volumes2 TiB and 8 TiB, each a ReadWriteOnce PVC mounted in its pod at /data, ext4 formatted
StorageClass (QoS)3,000 IOPS and 125 MiB/s (gp3 defaults for EBS)

backedblock.io

Storage cluster that hosts the tiered volumes, with hot data on EBS and cold data in S3. Each test volume gets a slice of the EBS cache, sized from the volume, and its own QoS limits.

Three nodes is the minimum for automatic failover. At this QoS (3,000 IOPS and 125 MiB/s per volume) the pool can hold 27 volumes, 9 per node, of either 2 TiB or 8 TiB: each node's EBS bandwidth caps it at nine volumes running at full rate. You can estimate the pool and the costs for your workload with the calculator.

Storage nodes3 × c6gn.4xlarge (16 vCPU, 32 GiB)
EBS bandwidth1,187.5 MB/s and 40,000 IOPS per node (instance limit)
Network25 Gbit/s per node
Cache disks10 × 460 GiB gp3 in RAID0 per node: 30,000 IOPS and 1,250 MB/s combined (3,000 IOPS and 125 MB/s each)
Volume cache size10% of the volume (max set to 465.7 GiB)
• 2 TiB volume: 204.8 GiB
• 8 TiB volume: 465.7 GiB
S3S3 Standard storage class, AES-256 encrypted, uncompressed

Test profiles

Every volume got its own test pod running fio, and every test pod ran on a separate node.

Each run uses a 128 GiB working set of 437 × 100 MiB, 43 × 1 GiB and 4 × 10 GiB files, written sequentially before the tests; every test runs on one of the four 10 GiB files. We ran the matrix three times on each volume, and by the last run it held 379 GiB. Every test is one fio run with the parameters from the table below:

bash
fio --name=<test> --rw=<pattern> --bs=<block size> \
    --iodepth=<queue depth> --numjobs=<jobs> \
    --filename=/data/<10 GiB file> --size=10G \
    --direct=1 --ioengine=libaio \
    --time_based=1 --runtime=60 --ramp_time=10 \
    --group_reporting=1 --randrepeat=0 --norandommap=1 \
    --output-format=json

The mixed tests add --rwmixread=70 or --rwmixread=50, and the durable write adds --fdatasync=1.

#TestPatternBlock sizeQueue depthJobs
1latency, readrandom read4k11
2latency, writerandom write4k11
3–7random readrandom read4k, 8k, 16k, 64k, 256k321
8–12random writerandom write4k, 8k, 16k, 64k, 256k321
13sequential readsequential read1M161
14sequential writesequential write1M161
15mixed 70/30random, 70% reads4k321
16mixed 50/50random, 50% reads16k321
17concurrent readrandom read4k328
18concurrent writerandom write4k328
19durable writerandom write, fdatasync after every write4k11

Queue depth 1 measures the latency of a single request; queue depth 32 puts enough requests in flight to reach the limits; 8 jobs × 32 (256 requests in flight) is a deliberate overload, to see how each side queues.

Results

A new volume is slightly slower while it's still initialising (cache ingestion, reads hit cache)

On a freshly formatted volume, ext4 zeroes its inode tables in the background (lazy init), a steady 8–12 MiB/s of writes the workload never issued. On backedblock.io those writes count against the volume's QoS budget, so while they run most tests land about 4–8% under the limit and single-request reads 27–37% under gp3 (p99 up to 10 ms on 8 TiB). It took about 1 hour on 2 TiB and 2 hours on 8 TiB, so the first run (the first 128 GiB set) ran on still-initialising volumes. The results below are from the second run, after init had finished.

Volumes initialised (cache has data, reads still hit cache)

  • backedblock.io holds its limits exactly. Every test bound by the limit ran at 2,999.0–2,999.7 IOPS or 124.9–125.2 MiB/s, on both sizes. gp3 ran up to 11% under its limit on 64k and 256k reads, and the 8 TiB volume about 15% under on 64k writes and about 7% under on sequential 1M.

  • Single-request and synced writes beat gp3 by 2.4–2.8×. QD1 random writes ran at 2,880–2,994 IOPS against about 1,060–1,080, and writes with fdatasync at 2,772–2,980 against 1,078–1,133. A write is acknowledged when it reaches the storage node's cache, which is built on ten gp3 disks in RAID0, so the writes are spread across all ten; the upload to S3 happens later.

       write:  pod ──▶ NVMe target ──▶ cache disk ──▶ ack          0.17–0.24 ms
                                           └────▶ S3, later
    
  • Single-request reads improved, but are still slightly slower than gp3. Read latency improved and mostly reaches gp3's values: the p99 fell to 0.92–0.95 ms, against 0.91–1.06 ms on gp3 (from 10.03 ms on the 8 TiB volume during lazy init). The median stays at 0.69 ms against 0.55 ms, so IOPS are 17% lower, because a read crosses the network to the storage node even when it hits the cache.

       read:   pod ──▶ NVMe target ──▶ cache hit  ──▶ ack          0.69 ms
                                     └─ miss ──▶ S3 GET ──▶ ack    ~120 ms
    
  • Under heavy load, backedblock.io has a much tighter tail than gp3. With 8 jobs × QD32, backedblock.io's p99 was 103–109 ms against 142–308 ms on gp3: 27–30% lower on reads, 40% lower on 2 TiB writes and 67% lower on 8 TiB writes.

  • On 2 TiB the cache reached its maximum size. Writing the second 128 GiB set filled the 204.8 GiB cache, so older data that nothing was reading was evicted to make room. Eviction did not affect the writes, which stayed at the limit, and every read in this run still hit the cache.

IOPS (higher is better):

Testgp3 2Tbackedblock.io 2TChange, 2Tgp3 8Tbackedblock.io 8TChange, 8T
QD1 randread 4k1,744.31,444.6-17.2%1,740.71,441.2-17.2%
QD1 randwrite 4k1,061.82,880.4+171.3%1,080.52,994.1+177.1%
randread 4k2,945.82,999.7+1.8%2,945.32,999.6+1.8%
randread 8k2,946.12,999.7+1.8%2,945.42,999.6+1.8%
randread 16k2,938.12,999.7+2.1%2,942.42,999.6+1.9%
randread 64k1,784.61,999.6+12.0%1,809.31,999.5+10.5%
randread 256k457.9499.8+9.2%468.8499.7+6.6%
randwrite 4k2,867.42,999.3+4.6%2,920.42,999.0+2.7%
randwrite 8k2,946.82,999.2+1.8%2,918.32,999.2+2.8%
randwrite 16k2,999.12,999.20.0%2,904.62,999.3+3.3%
randwrite 64k2,002.71,999.5-0.2%1,708.81,999.5+17.0%
randwrite 256k500.3499.7-0.1%464.9499.8+7.5%
seq read 1M125.1124.9-0.2%117.0124.9+6.8%
seq write 1M125.1124.9-0.2%116.5124.9+7.2%
randrw 70/30 4k2,999.12,999.10.0%2,957.92,999.2+1.4%
randrw 50/50 16k2,999.12,999.20.0%2,955.12,999.2+1.5%
8 jobs × QD32 randread 4k2,999.12,999.80.0%2,951.72,999.5+1.6%
8 jobs × QD32 randwrite 4k2,999.22,998.50.0%2,940.72,999.2+2.0%
QD1 write + fdatasync 4k1,132.72,772.2+144.7%1,078.12,979.6+176.4%

Latency, p50 / p99 in ms (lower is better):

Testgp3 2Tbackedblock.io 2TChange, 2Tgp3 8Tbackedblock.io 8TChange, 8T
QD1 randread 4k0.55 / 1.060.69 / 0.92+25.1% / -12.8%0.55 / 0.910.69 / 0.95+25.1% / +4.5%
QD1 randwrite 4k0.92 / 1.520.17 / 0.74-82.1% / -51.1%0.93 / 1.070.24 / 0.70-74.5% / -34.8%
randread 4k10.42 / 11.8610.03 / 11.21-3.8% / -5.5%10.42 / 11.8610.03 / 11.21-3.8% / -5.5%
randread 8k10.42 / 11.7310.16 / 11.60-2.5% / -1.1%10.42 / 11.8610.16 / 11.60-2.5% / -2.2%
randread 16k10.42 / 11.9910.03 / 11.60-3.8% / -3.3%10.42 / 11.9910.03 / 11.47-3.8% / -4.4%
randread 64k16.91 / 26.8715.93 / 16.91-5.8% / -37.1%16.91 / 22.9415.93 / 17.17-5.8% / -25.1%
randread 256k65.80 / 82.3162.65 / 64.75-4.8% / -21.3%66.32 / 76.0262.65 / 64.75-5.5% / -14.8%
randwrite 4k10.68 / 13.8310.03 / 13.57-6.1% / -1.9%10.68 / 12.3910.03 / 11.08-6.1% / -10.6%
randwrite 8k10.03 / 13.7010.03 / 13.570.0% / -1.0%10.68 / 12.3910.03 / 11.21-6.1% / -9.5%
randwrite 16k10.03 / 11.3410.03 / 13.440.0% / +18.5%10.55 / 12.7810.03 / 11.21-5.0% / -12.3%
randwrite 64k15.01 / 16.7115.01 / 18.480.0% / +10.6%16.45 / 24.7715.01 / 16.91-8.8% / -31.7%
randwrite 256k60.03 / 67.6363.18 / 65.80+5.2% / -2.7%67.63 / 83.3663.18 / 63.18-6.6% / -24.2%
seq read 1M120.06 / 149.95120.06 / 121.110.0% / -19.2%127.40 / 177.21120.06 / 121.11-5.8% / -31.7%
seq write 1M120.06 / 173.01120.06 / 123.210.0% / -28.8%127.40 / 185.60120.06 / 121.11-5.8% / -34.7%
randrw 70/30 4k10.16 / 11.3410.55 / 13.44+3.9% / +18.5%10.42 / 11.7310.16 / 11.60-2.5% / -1.1%
randrw 50/50 16k10.29 / 11.3410.68 / 13.57+3.8% / +19.7%10.42 / 11.7310.29 / 11.73-1.3% / 0.0%
8 jobs × QD32 randread 4k81.27 / 154.1482.31 / 107.48+1.3% / -30.3%82.31 / 141.5682.31 / 103.280.0% / -27.0%
8 jobs × QD32 randwrite 4k76.02 / 179.3182.31 / 108.53+8.3% / -39.5%74.97 / 308.2882.31 / 103.28+9.8% / -66.5%
QD1 write + fdatasync 4k0.87 / 0.950.16 / 0.51-81.3% / -47.0%0.93 / 1.070.16 / 0.41-82.7% / -61.6%

What a cache miss costs (cache miss on ~10% of reads)

Between the second and third runs, 128 GiB of other files were written to both volumes. The 2 TiB volume's cache was already full, so those writes evicted part of the test files.

  • A read that misses the cache goes to S3. It fetches the 1 MiB block containing the read, which took about 100–120 ms, against well under a millisecond for a hit.

       read:   pod ──▶ NVMe target ──▶ cache hit  ──▶ ack          0.69 ms
                                     └─ miss ──▶ S3 GET ──▶ ack    ~120 ms
    
  • A few percent of misses set the tail. On 2 TiB, 7–13% of random reads missed, and the p99 of every affected test became the S3 round trip: 142–146 ms, against 1–27 ms on gp3.

  • With one request at a time, misses set the average too. QD1 reads fell to 110 IOPS, against 1,744 on gp3. At queue depth 32 each miss holds a slot for about 120 ms, so 4k–64k random reads landed 4–31% under gp3.

  • Writes are not affected. They always land in the cache first, so single-request writes still beat gp3 by 2.7–2.8× and synced writes by 2.6–2.7×.

  • The cache refills as data is read. The 16k test read the same file as the QD1 test before it, whose misses had already brought the blocks back: 0.09% misses, at the limit.

Only the tests where misses hit backedblock.io are shown; every other test, and every test on 8 TiB, stayed about where it was in the previous run. 2 TiB volume:

IOPS (higher is better):

Testgp3backedblock.ioChangeReads that missed
QD1 randread 4k1,744.3110.2-93.7%worst case: reads are consecutive, so each miss stalls the next read
randread 4k2,945.82,522.5-14.4%7.9%
randread 8k2,946.12,030.9-31.1%7.3%
randread 64k1,784.61,720.4-3.6%13.0%

Latency, p50 / p99 in ms (lower is better):

Testgp3backedblock.ioChange
QD1 randread 4k0.55 / 1.060.70 / 145.75+26.6% / +13689.2%
randread 4k10.42 / 11.868.45 / 141.56-18.9% / +1093.4%
randread 8k10.42 / 11.739.50 / 145.75-8.8% / +1142.5%
randread 64k16.91 / 26.8711.99 / 145.75-29.1% / +442.4%

What a cold start costs (cache empty, data only in S3)

The worst case is a volume with nothing cached at all. That is exactly what a restore from a snapshot gives you: the data is in S3 until something reads it. We ran this one on a 2 TiB volume restored from a snapshot; nothing in it depends on the volume size, because a cold read costs the same per block whatever the volume. We read one 10G file per phase.

  • A cold volume is slow until its data is warm. A cold random read is an S3 round trip, 121 ms on average.
  • Each missed read warms the cache with a whole 1 MiB block, which then serves the other 255 4k offsets inside it, so warming speeds up as it goes. At queue depth 32 a cold 10 GiB file reached the 3,000 IOPS limit in 46 seconds and about 9,800 S3 GETs, one per block.
  • Sequential reads on a cold volume are much faster. Readahead fetches ahead of the stream and warms the cache before the reads arrive: a cold 1M read ran at 88 MiB/s in its first second and at 125 MiB/s from the second on, 123 MiB/s on average against 125 warm.
  • The worst case is random reads that miss one at a time. At queue depth 1 each miss stalls the next read, so a cold volume ran at 8 IOPS, about 175× slower than the same read once the data is cached.
  • Pre-warm the hot data after a restore. Reading the hot files once sequentially (cat, dd or vmtouch in an init container) ran at the throughput limit, 82 seconds per 10 GiB here, and random reads on that file were at their limit from the first second afterwards. Once warm, the restored volume matched the original: 1,427 IOPS and a 0.98 ms p99 on QD1 reads, against 1,445 and 0.92 ms.

Each phase:

PhaseIOPSMiB/sp99Misses
Cold, QD1 4k random8 (worst case: random reads always miss, and each miss stalls the next read)0.03225 ms96%
Cold → warm, QD32 4k random, 600 s2,826 avg, at the limit from 46 s11.011.1 ms0.6%
Cold, sequential 1M123123234 ms8.9%
After a sequential pass, QD32 4k random3,000 from the first second11.711.1 ms0%
Warm, QD1 4k random1,4275.60.98 ms0%

The first 46 seconds of the QD32 phase, as the cache warms on a cold 10 GiB file:

SecondIOPSMean latency
1200~135 ms
10286111 ms
2034491 ms
3058854 ms
401,18926 ms
462,99610 ms

What it costs on the bill

The backedblock.io has a fixed cost that gp3 does not (storage nodes and their cache disks) and a variable cost that is much lower, because S3 charges for bytes written and gp3 for bytes provisioned. So the answer depends on how much capacity there is to spread the fixed part over.

Monthly, backedblock.io / gp3, and the saving against gp3 (negative where backedblock.io costs more), for 2 TB volumes:

VolumesCapacity onlySavingWith snapshotsSaving
1$836 / $160-422.4%$925 / $420-120.2%
5$1,152 / $800-43.9%$1,598 / $2,100+23.9%
10$1,546 / $1,600+3.4%$2,438 / $4,200+41.9%
25$3,487 / $4,000+12.8%$5,717 / $10,500+45.6%
50$6,956 / $8,000+13.1%$11,416 / $21,000+45.6%
100$13,912 / $16,000+13.1%$22,832 / $42,000+45.6%

And for 8 TB volumes:

VolumesCapacity onlySavingWith snapshotsSaving
1$1,033 / $640-61.4%$1,390 / $1,680+17.3%
5$2,139 / $3,200+33.2%$3,923 / $8,400+53.3%
10$3,773 / $6,400+41.1%$7,341 / $16,800+56.3%
25$8,422 / $16,000+47.4%$17,342 / $42,000+58.7%
50$16,844 / $32,000+47.4%$34,684 / $84,000+58.7%
100$33,689 / $64,000+47.4%$69,369 / $168,000+58.7%

Snapshots are one full and three incrementals a month, kept for six months, in S3 Glacier Instant Retrieval on backedblock.io and as EBS snapshots on gp3. These figures price the storage nodes at On-Demand rates. Reserved Instances or a Savings Plan lower that compute cost, and with it the fixed part of the pool, so the savings can go further.

For capacity alone, the break-even point is set by total capacity rather than by the number of volumes: about ten 2 TB volumes or two to three 8 TB volumes, roughly 16–24 TB in the pool either way. Below that, gp3 is cheaper, and for a single small volume it is not close. With snapshots it comes much earlier: five 2 TB volumes already save about 24%, and a single 8 TB volume about 17%.

Past it, the volume size decides how much is saved. With 2 TB volumes the saving levels off at about 13%, or 46% with snapshots, and with 8 TB volumes at about 47%, or 59% with snapshots. Every volume carries some compute of its own, and a larger volume spreads it over more bytes.

Try it

You can estimate the costs and savings for your own workload with the calculator, then deploy backedblock.io in your AWS account and test it on your workload: see Get started.

← All posts