Pool node management

Taking a storage node out of service and putting it back is two commands. What happens in between is worth knowing before the first time you need it — draining a node is not a no-op that only stops new placements, it empties the node.

text

scheduler node drain <node>
scheduler node uncordon <node>

Both are CLI operations against any node's API, and both are idempotent. The same machinery runs unattended when a node fails on its own — see Failover.

#What draining does

Marks the node system_drain=yes, so the scheduler stops placing new volumes on it.
Stops the volumes running on it, waiting for each to report that it has stopped.
Re-stamps those volumes so the scheduler places them again. They land on the survivors, because the drained node is no longer a candidate.
Once nothing references the node — no volume assigned, no peer holding its mount, SPDK unmounted — hands its cache array to a surviving node: the cache volumes are detached and re-attached to that node's instance.

The array and the volumes that live on it travel together, and they come back together on uncordon.

#What it costs


Volume restart	Every volume on the node stops and starts elsewhere. I/O pauses for that gap — this is not a live migration.
Cold cache	The volumes arrive with an empty cache and warm up from object storage, so latency is worse until their working sets are resident again — see Tradeoffs.
Room on the survivors	They need free volume slots and free cache budget. A pool sized so that every node is exactly full has nowhere to put the evacuated volumes, and they will sit unplaced until it does.
Lost capacity	The pool is down one node's cores, cache, IOPS and bandwidth for the duration, so per-volume headroom shrinks while the work does not.
Quorum	Three nodes tolerate one being away. Draining a second at the same time is the outage, not the maintenance — see High availability.

#Draining for maintenance

Check there is room. The survivors have to absorb this node's volumes: max_volumes_count gives the slots and the two cache budgets give the space. Sizing the pool prints both for a given workload.
Drain it. scheduler node drain <node>.
Wait for it to empty. The node is clear when its volumes are running elsewhere and its array has been handed over. Until then the pool is still moving data around, and a reboot mid-move only makes the next loop redo it.
Do the work — reboot, resize, patch, replace the instance.
Bring it back. scheduler node uncordon <node>. The node reclaims its own array once the peer has released it, the cache volumes move home, and the volumes that belong to that array are rescheduled to it.
Expect a warm-up. Everything that moved, in both directions, starts cold.

Nothing in that sequence is on a timer, so a node can sit drained as long as the maintenance takes.

#Upgrading a node

An instance-type change or a new node AMI is the same procedure with Terraform in the middle: drain the node, apply the change for it, uncordon. Doing it one node at a time is what keeps the pool serving throughout — with three nodes there is exactly one node of slack, so a rolling upgrade is serial by definition.

Growing a pool is the easier direction and needs none of this: a new node is a terraform apply, and the scheduler starts using it for the next volume it places. It does not move existing volumes onto it — see Volume scheduling.

#When a node fails instead

The steps above are the deliberate version of what the pool does by itself after a node has been unreachable for five minutes. The difference is only in step 5: a failed node is never uncordoned automatically, so it stays out of the pool until someone decides it is healthy.