Architecture
backedblock.io is an implementation of the disaggregated block storage pattern, with object storage as the capacity tier. Each volume gets its own network target, its own cache and its own set of objects in a bucket, and all run on storage nodes rather than on whichever node the workload landed on. Kubernetes is where you drive it: the driver extends Kubernetes' own APIs, so provisioning, expansion, snapshots and restores are ordinary PVC operations.
Kubernetes cluster storage cluster
┌─────────────────────────┐ ┌─────────────────────────┐
│ CSI Driver │──── HTTP/s ────▶│ management API │ ┐ control plane
│ │ │ │ ┘
├─────────────────────────┤ ├─────────────────────────┤
│ pod │ │ NVMe target (SPDK) │ ┐
│ └─ /dev/nvmeXnY │◀─ NVMe-oF/TCP ─▶│ nbdkit plugin │ │ data plane
│ │ │ cache on local disks │ │
│ │ │ │ │ ┘
└─────────────────────────┘ └────────────┼────────────┘
▼ 1 MiB objects
┌───────┐
│ S3 │
└───────┘
The two planes are worth keeping apart in your head. The control plane creates, attaches, expands and snapshots volumes; the data plane carries blocks. They run in different places and fail independently — a control plane that is down cannot provision anything, but it does not stop a mounted volume from serving I/O.
They also live on opposite sides of a boundary. Kubernetes runs the CSI driver and nothing else: no capacity, no cache, no part of the data path. The storage cluster owns the disks, the buckets and the volumes, and is deployed outside Kubernetes.
Control plane
The control plane is the set of components responsible for:
- Managing storage cluster metadata and operations
- Placing and configuring volumes
- Interfacing with CSI to manage the lifecycle of a volume
- Interfacing with CSI for the operations around it — snapshots, restores, expansion, cleanup
- Exporting telemetry to Prometheus and Grafana
- Providing the surfaces you debug and operate from: an API, a CLI and logs
Nothing in it touches your data. A PVC becomes an API call, and what comes back is the address of a target to connect.
┌──────────────┐ CSI gRPC ┌────────────────────┐ management ┌──────────────┐
│ kube-api │──────────────▶│ mgxcsi-controller │──────────────▶│ storage │
│ │ │ (provision/snap) │ API, HTTP/s │ cluster │
└──────────────┘ └────────────────────┘ └──────┬───────┘
│ NVMe-oF
┌──────────────┐ CSI gRPC ┌────────────────────┐ │
│ kubelet │──────────────▶│ mgxcsi-node │◀─────────────────────┘
│ (mount) │ │ (connect/mount) │ /dev/nvmeXnY
└──────────────┘ └────────────────────┘
Everything is driven through the API. It is HTTP/S and it is the same surface for every caller — the CSI driver, your own automation, or a person at a terminal. Two ways to speak to it:
- JSON RPC
- Declarative YAML manifests with
metadataandspecsections, for people who would rather keep the desired state in a file and apply it.
A CLI ships on the nodes for the operator's side of that: inventory a cluster, look at nodes and volumes, and drive the same operations by hand.
Note. Small installations do not need separate control-plane nodes. The control-plane and data-plane components can run together on the same machines, which collapses the whole deployment into N equivalent nodes forming one storage pool — every node identical, every node running both planes. Larger deployments split them, with a management plane of its own and pools beneath it.
Data plane
The data plane is the set of components responsible for:
- Serving each volume to the workload as an ordinary block device, over NVMe-oF/TCP
- Enforcing the per-volume QoS
- Absorbing writes into the local write-back cache
- Answering reads from cache, and fetching missed blocks from S3
- Compressing and encrypting blocks before they leave the node
The path a block takes
┌─────────────────────────────┐
│ pod: ext4 on /dev/nvmeXnY │
└──────────────┬──────────────┘
write ─▶ │ ◀─ read NVMe-oF/TCP
┌──────────────▼──────────────┐
│ NVMe target │ per-volume MB/s and IOPS caps
└──────────────┬──────────────┘
┌──────────────▼──────────────┐
│ write-back cache │ a write is acknowledged here
└──────────────┬──────────────┘
flush ─▶ │ ◀─ miss 1 MiB blocks
┌──────────────▼──────────────┐
│ S3 bucket │ every block, durably
└─────────────────────────────┘
A write is acknowledged once it is in the cache and flushed to the bucket in the background, coalesced into whole 1 MiB blocks so a page of small random writes does not become a page of uploads. A read is served from the cache if the block is resident, and otherwise fetches that one 1 MiB block first.
The layers
| Layer | What it does |
|---|---|
| Kubernetes | The CSI driver: a controller that provisions, expands and snapshots volumes, and connects the target and mounts the device |
| Target | An SPDK NVMe-oF/TCP subsystem per volume. The volume is exposed over the network and reachable from the workload nodes |
| Cache | nbdkit plugin, serving the block API and owning a cache file on local disk. It absorbs client writes and serves hits without touching S3 |
| Object storage | nbdkit plugin, flushing aggregated blocks out in the background as 1 MiB objects, applying compression and encryption on the way, and fetching a block back from S3 on a read miss |
What gets deployed
Everything runs in your AWS account and an existing VPC — normally the same VPC your workloads already run in, so the CSI driver reaches it over private addresses with nothing to open. The cloud resources the stacks create:
┌─ AWS account ──────────────────────────────────────────────────────────────┐ │ │ │ ┌─ your existing VPC ────────────────────────────────────────────────────┐ │ │ │ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────────┐ │ │ │ │ │ ec2 │ │ ec2 │ │ ec2 │ │ bastion │ │ │ │ │ │ node 1 │ │ node 2 │ │ node 3 │ │ (optional) │ │ │ │ │ └┬─┬────┬──┘ └┬─┬────┬──┘ └┬─┬────┬──┘ └────────────┘ │ │ │ │ │ │ │ │ │ │ │ │ │ ssh jump host │ │ │ │ │ │ ┌──┴────┐ │ │ ┌──┴────┐ │ │ ┌──┴────┐ public subnet │ │ │ │ │ │ │ EBS │ │ │ │ EBS │ │ │ │ EBS │ │ │ │ │ │ │ │ cache │ │ │ │ cache │ │ │ │ cache │ EBS cache │ │ │ │ │ │ └───────┘ │ │ └───────┘ │ │ └───────┘ disks per node │ │ │ │ │ │ │ │ │ │ │ │ │ │ ────┴─┼─────────────┴─┼─────────────┴─┼─ mgmt subnet · HTTP/s API │ │ │ │ │ │ │ │ │ │ │ ──────┴───────────────┴───────────────┴─ data subnet · NVMe-oF/TCP │ │ │ └────────────────────────────────────┬───────────────────────────────────┘ │ │ │ S3 gateway endpoint │ │ ▼ │ │ ┌─ S3 ─────────────────────────────────────────────┐ │ │ │ ┌──────────────────┐ ┌────────────────────┐ │ │ │ │ │ data buckets │ │ backup buckets │ │ │ │ │ └──────────────────┘ └────────────────────┘ │ │ │ └──────────────────────────────────────────────────┘ │
Those EC2 nodes, their cache disks and their buckets are one pool — the unit volumes are placed in and served from.
- Two subnets per AZ, one network each way. The mgmt subnet carries
control traffic — the management API on
8082, ssh, metrics — and the data subnet carries volume I/O over NVMe-oF/TCP. Every node has an interface in both. - EC2 storage nodes, each with its own cache disks — EBS volumes striped RAID0. That cache is what gives a volume its latency, so its size is the number worth getting right before you apply.
- Data and backup buckets — the data buckets hold every block as a 1 MiB object, the backup buckets hold snapshots. Both are reached through an S3 gateway endpoint, so block traffic never leaves the AWS network.
- A NAT gateway, a security group, and an optional bastion in a public subnet you provide — the ssh jump host for nodes that have no public IP.
What the nodes run
Every node boots a prebaked AMI with all the applications a storage cluster
needs already compiled and installed. Terraform installs no software: it
delivers the per-node inputs (secrets, peer addresses, the pool registry) and
runs the baked setup-node.sh in place.
One image, both planes. The AMI is shared across the whole deployment, so
provisioning is mostly a choice of role — mgmt for a control-plane node,
pool for a data-plane node, or both on the same machine for simple
single-pool setups.
Base OS, architecture and the published AMI ID for each region: Node AMIs.
The same stack runs on both roles; only the plugin set differs.
control-plane node · role mgmt data-plane node · role pool ┌────────────────────────────────┐ ┌────────────────────────────────┐ │ HTTP/S API · JSON RPC :8082 │────────▶│ HTTP/S API · JSON RPC │ ├────────────────────────────────┤mgmt cmds├────────────────────────────────┤ │ core │ │ core │ │ joins nodes · elects leader │ │ joins nodes · elects leader │ │ keeps the cluster running │ │ keeps the cluster running │ ├────────────────────────────────┤ ├────────────────────────────────┤ │ plugins │ │ plugins │ │ management operations API │ │ volume lifecycle · QoS │ │ pool registry · federation │ │ NVMe target · cache · S3 │ ├────────────────────────────────┤ ├────────────────────────────────┤ │ cassandra · cluster metadata │ │ cassandra · cluster metadata │ ├────────────────────────────────┤ ├────────────────────────────────┤ │ prometheus (optional) │ │ prometheus (optional) │ └────────────────────────────────┘ └────────────────────────────────┘
Applications
- core — the distributed application every node runs. It joins nodes into a cluster, elects a leader, self-heals and keeps the cluster up. Each group of nodes is a self-sufficient pool with its own membership and its own leader, and does not depend on any other pool.
- plugins — core extensions, each responsible for one piece of functionality. Depending on the node's role core spawns a different set of them: management operations and the pool registry on a control-plane node, volume lifecycle, QoS and the data path on a data-plane node. That set is the only difference between the two roles.
- Cassandra — the persistence layer for cluster metadata.
- HTTP/S API — core exposes JSON RPC covering every cluster operation, the same surface for every caller: the CSI driver, your own automation, or a person at a terminal. It is also how a control-plane pool propagates management commands down to the data-plane pools. The CLI on each node drives that same surface.
Metrics (optional)
A node can be started with the metrics feature enabled (enable_metrics), which
brings up Prometheus on the node to collect and aggregate metrics from core and
its plugins. enable_grafana adds Grafana on the cluster's VIP node.