Storage compute

A storage node is not a disk shelf. It runs the whole data path in software — the NVMe-oF target, the cache, the S3 client — so nodes_instance_type is a real choice: it sets how much volume traffic a node can carry, not just how much it costs.

Sizing a pool right now? Sizing the pool runs the arithmetic on this page for you — cores, memory, network and EBS bandwidth for a given volume size, count and throughput, the volumes per node and max_volumes_count that come out cheapest, and the instance type and node count they add up to.

#What is running on a node

#What actually limits a node

• Network. Every read and write crosses the network twice in the worst case: NVMe-oF/TCP from the workload, and S3 for a miss or a flush. The instance's bandwidth is the ceiling on both, and they share it.
• EBS throughput, when the cache is EBS. The instance's EBS bandwidth caps what the cache tier can absorb, regardless of how many volumes you stripe.
• CPU. SPDK polls, and compression and encryption happen on the node before a block leaves it. Neither is free at line rate.
• Memory. The 4 GiB of hugepages, plus Cassandra, plus the cache's own bookkeeping.

#Node count and instance type

The limits above are what to size against; this is how they turn into an instance type and a node count. 4 vCPU and 16 GiB is the minimum for a storage node, and it is worth knowing where that goes before choosing anything larger.

CPU. SPDK takes two cores and keeps them — it polls, so they are dedicated rather than shared. What is left carries two very different things:

• mgx-core with its plugins, and Cassandra. Neither is CPU-hungry; they do not sit in the data path.
• One nbdkit process per attached volume, which does. Reckon on about half a core per volume on average, more with encryption or compression enabled and more under sustained load. It is an average across instance types and CPU architectures rather than a constant, so measure it on the instance you actually chose and revisit the number as the pool fills.

Memory. SPDK's 4 GiB of hugepages, plus Cassandra and the OS, plus a per-volume share that tracks the provisioned size of the volume rather than how much of it is in use: each nbdkit process allocates internal structures for it — the block bitmap and its friends — and leans on the filesystem cache for dirty write blocks. On average that is about 1.5 GiB for a 1T volume and about 3.5 GiB for a 16T volume. Like the CPU figure it wants measuring, and it wants re-measuring over the pool's life rather than once at design time.

Network and EBS bandwidth. Both are per-instance ceilings, and the same work crosses both. A volume's throughput arrives over NVMe-oF/TCP in full, and part of it leaves again towards S3 — only the part that misses or flushes, because a read served from cache never leaves the node. Doubling the client figure is therefore the worst case rather than the working one; reckon the S3 side at about half, and size for that. The cache reads and writes the traffic in the middle, so EBS carries the one-way figure. Five volumes at 200 MB/s each is 1000 MB/s of client traffic:

network   1000 MB/s in from NVMe-oF/TCP  +  ~500 MB/s out to S3   ≈ 12 Gbit
EBS       1000 MB/s of cache writes and reads

For example, an m8gb.2xlarge clears both — 16.7 Gbit of network and 1562 MB/s to EBS — with the margin going to bursts and to warm-up, when a cold volume reads from S3 and writes the cache at the same time. The xlarge below it does not, at 8.3 Gbit and 781 MB/s. Both figures are AWS baseline rather than burst, which is the basis to size on.

Neither ceiling is a budget anything enforces. Place more volumes on a node than the ceilings support, or hand out QoS caps that add up past them, and the node saturates: throughput flattens and latency rises for every volume on it rather than only the busy one. That is the failure mode overprovisioning has to be watched for, and it is not visible in the volume's configuration — only in what the node is doing.

Put together, a node's own arithmetic:

cores   = 2 (SPDK) + ~0.5 * volumes per node + ~1 (core, plugins, Cassandra)
memory  = 4 GiB (hugepages) + ~2 GiB * volumes per node + ~4 GiB (OS, Cassandra)
network = volumes per node * qos_rw_mbytes_per_sec * 1.5    # client side + ~half again for S3
EBS     = volumes per node * qos_rw_mbytes_per_sec

The lever in every line is volumes per node, which is what max_volumes_count sets — so the sizing runs from the volumes you intend to place, not from the instance catalogue. Given a volume size and a count: pick how many volumes should share a node, read the four figures off the lines above, choose the instance that clears all of them, then take the node count as total volumes / volumes per node — never fewer than three, for the reasons in the next section.

Which makes nodes_instance_type and nodes_count a real choice rather than a default to inherit. The same fifteen volumes fit on three large nodes or five small ones, and the two pools behave differently: fewer nodes concentrate the bandwidth ceilings and make one node's loss expensive, more nodes spread both and cost more per volume in overhead.

Turn on enable_metrics and enable_grafana when the pool is applied and all of it becomes observable rather than assumed. Per-volume CPU and RSS, and the node's network and EBS throughput against its ceilings, are what to watch; any of them can move you a size up, or show there is room to place more volumes per node than you first guessed.

#High availability

Three storage nodes is the floor, whatever the capacity arithmetic says. It is not a performance number — it is what the pool needs to keep its own state and to survive losing a node.

• The metadata store is replicated three ways. A node's bootstrap creates the pool's keyspace with NetworkTopologyStrategy and replication_factor: 3, so three nodes are what it takes to hold one copy of the metadata each. Fewer, and replicas have nowhere to go.
• Quorum is floor(n / 2) + 1. At three nodes that is two, so the pool keeps answering with one node down. At two nodes quorum is also two, so nothing may be lost at all; at one there is no quorum to speak of.
• A lost node's volumes have to land somewhere. The survivors take over the drained node's cache array, which needs both the volume slots and the cache budget to be there — so a pool sized to exactly full on every node has no failover room. See Volume scheduling.

Terraform does not enforce this: nodes_count validates only as >= 1, and a one-node pool will come up and serve volumes. It is a lab, not a deployment.

Above three, node count is a capacity and throughput decision like any other — that is the next section.

#Failover

Failover is automatic, and the thing that moves is not a volume on its own — it is a node's whole cache array, with the volumes that live on it.

The trigger is the heartbeat. A node reported DOWN for more than 300 seconds is labelled system_drain=yes automatically — the same mark scheduler node drain sets by hand — and the scheduler stops considering it from then on. Five minutes is deliberately long: a reboot or a network blip should not shuffle a pool.

One node runs the pass. The pool's VIP holder handles failover for the whole cluster; every other node only ever stops its own volumes. A node that finds itself in a minority partition stops its volumes locally without waiting to be told, and the majority side deals with its own half independently.

Then, in order:

1. The failed node's volumes are re-stamped back to the start of the placement path, so the scheduler places them again — on the survivors, since the failed node is out of the candidate set by now.
2. Once nothing references that node any more — no volume assigned to it, no peer holding its mount, SPDK unmounted — its cache array is shifted to a surviving node. That shift is physical: the EBS volumes are detached and re-attached to the new host's instance.
3. The receiving node records the failed node's array alongside its own, which raises its volume cap and its cache budget by one array's worth — both are per array. It still has to have the room, which is the argument for not sizing every node exactly full.

Coming back is manual. Nothing removes the drain mark on its own: scheduler node uncordon <node> does. Only then is the node's own array brought home — the peer releases it, the EBS volumes move back, and the volumes that live on that array are rescheduled to it. A node that dropped out once is worth looking at before it takes data again, which is why this step is a decision rather than a timer.

Volumes that moved arrive with a cold cache and warm up from object storage the same way a new one does — see Tradeoffs. The whole procedure, as an operation you run on purpose, is Pool node management.

#Scaling it

Two dials, and they do different things:

• nodes_instance_type — a bigger node carries more throughput per volume and more volumes before the cache budget runs out.
• nodes_count — more nodes in the pool is more aggregate throughput and more room, and the scheduler spreads volumes across them. Adding a node is a terraform apply; the pool absorbs it.

Neither moves the volumes already placed. To empty a node before resizing it, drain it first — see Volume scheduling.

#Management nodes

A control-plane node runs no data path: no SPDK target, no cache, no S3 traffic. It carries the API, the metadata store and the pool registry, so it is sized for the control side alone and is typically a much smaller instance than a storage node.