Provisioning volumes

A volume is a PVC, and everything about it — how much cache it gets, its IOPS and bandwidth ceiling, whether it is compressed or encrypted, which pool it lands in — comes from the StorageClass the claim names. Provisioning, snapshotting and deletion are all ordinary Kubernetes operations; there is no separate control plane to drive by hand.

#Tiers are StorageClasses

Storage is rarely one thing. A database wants a large cache and no QoS ceiling in its way; a log archive wants compression and a cap so it cannot take the pool's drain capacity from anyone else; a tenant under a compliance obligation wants encryption, and possibly hardware nobody else is on.

Those are tiers, and the way you express a tier in Kubernetes is a StorageClass. Each one is a fixed profile — the parameters are read at creation and baked into the volume — and a workload opts into a tier by naming the class in its claim. Nothing else in the manifest changes.

Three tiers over the same cluster:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mgxcsi-sc-fast          # OLTP: big cache, no ceiling
provisioner: csi.migrx.io
parameters:
  labels: "tier=fast"
  ratio_cache_r_cache_size: "0.3"
  ratio_cache_rw_cache_size: "0.15"
  min_cache_r_cache_size: "8192"
  min_cache_rw_cache_size: "4096"
  max_cache_r_cache_size: "131072"
  max_cache_rw_cache_size: "16384"
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mgxcsi-sc-standard      # the default shape, with a ceiling
provisioner: csi.migrx.io
parameters:
  labels: "tier=standard"
  ratio_cache_r_cache_size: "0.1"
  ratio_cache_rw_cache_size: "0.05"
  max_cache_r_cache_size: "20480"
  max_cache_rw_cache_size: "3072"
  qos_rw_ios_per_sec: "20000"
  qos_rw_mbytes_per_sec: "500"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mgxcsi-sc-archive       # cold and compressed, capped hard
provisioner: csi.migrx.io
parameters:
  labels: "tier=archive"
  ratio_cache_r_cache_size: "0.02"
  ratio_cache_rw_cache_size: "0.01"
  max_cache_r_cache_size: "4096"
  max_cache_rw_cache_size: "1024"
  qos_rw_mbytes_per_sec: "100"
  storage_compress: "6"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

Every parameter, with its Helm value and default, is in StorageClass parameters. Existing volumes keep what they were created with — editing a class only changes volumes created after it.

#Binding a class to a pool

The three classes above differ in software. Where a tier has to differ in hardware — io2 cache disks rather than gp3, a particular AZ, a pool no other tenant is on — the class has to reach a specific pool, and that is done with labels.

Behind one StorageClass there may be one pool or several. A class is not bound to a pool directly; it selects a set of them, and the scheduler picks which one a given volume goes to. Three labels, matched in a chain:

  StorageClass parameters          scheduler config            pool (Terraform)
  ┌──────────────────────┐        ┌──────────────────┐        ┌──────────────────┐
  │ labels: "tier=fast"  │───────▶│ labels:          │───────▶│ labels =         │
  │                      │ volume │   "tier=fast"    │  pool_ │  "tier=fast,..." │
  │                      │ labels │ pool_selector:   │selector│                  │
  │                      │  match │   "tier=fast"    │ matches│                  │
  └──────────────────────┘        └──────────────────┘        └──────────────────┘
        the volume                  the placement policy         one or more pools

1. The class's labels are stamped on the volume at creation.
2. The scheduler picks the config whose own labels share a key=value with the volume's. No match and the config marked default is used instead.
3. That config's pool_selector matches the pool labels set in the pool's Terraform, and only matching pools are candidates.

So label the pools first:

# pools/io2-a/terragrunt.hcl
labels = "tier=fast,az=us-east-1a"

# pools/gp3-a/terragrunt.hcl
labels = "tier=standard,az=us-east-1a"

# pools/gp3-b/terragrunt.hcl
labels = "tier=standard,az=us-east-1b"

then add one scheduler config per tier from the CLI:

mgx-core:127.0.0.1:main:main:admin> scheduler config add --name fast \
    --enabled yes --labels tier=fast --pool_selector tier=fast
mgx-core:127.0.0.1:main:main:admin> scheduler config add --name standard \
    --enabled yes --labels tier=standard --pool_selector tier=standard \
    --load_strategy LeastAllocated

mgxcsi-sc-fast now lands on io2-a and nothing else. mgxcsi-sc-standard has two pools behind it, and LeastAllocated spreads volumes across both — adding a third tier=standard pool widens the tier with no change to the class and no change to any workload. The capacity guards and the spread-versus-pack choice are in Volume scheduling.

Isolation works the same way. Give a tenant's pool labels = "tenant=acme", a scheduler config that selects it, and a StorageClass that carries the matching volume label — their volumes are then on their nodes, their cache disks and their buckets, and no other class can reach them.

#The S3 backend

Each pool owns exactly two buckets: a data bucket, holding every block of every volume in the pool as a 1 MiB object, and a backup bucket for its snapshots. They are s3_bucket_names and s3_backup_bucket_names in the pool's Terraform — one name in each — and the pool's apply creates them.

#One prefix per volume

Inside the data bucket, every volume gets a prefix of its own — its name:

s3://mgxs3storage-gp3a/
├── vol-1/     ← volume "vol-1" — its blocks, and nothing else
├── vol-2/
└── vol-3/

That is why one bucket is enough. S3 scales request rate per prefix, not per bucket — 3,500 writes and 5,500 reads per second each, and it splits partitions further as load grows — so a pool's whole volume population sits behind one bucket without contending for its request rate.

The prefix is also what makes deletion tractable: purging a volume is purging one prefix, and nothing else in the bucket is ever in scope.

#Moving a volume to another pool

A volume does not leave the pool it was created in. The pool is stamped on the volume the moment it is placed, and it is never rewritten: every placement decision after that — a drain, a node failure — is constrained to nodes in that same pool.

So moving a volume between pools is snapshot and restore, and only that. The data is copied: the restore reads the source pool's backup bucket and writes a fresh prefix into the target pool's own data bucket, so what comes out is an independent volume, in the target pool, in the target pool's bucket. That is the route across AZs, and the way to promote a volume onto faster hardware — snapshot it, then restore with a storageClassName whose tier resolves to the pool you want.

# vol-1 is in a tier=standard pool. Restore it into the tier=fast one.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vol-1-fast
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: mgxcsi-sc-fast     # the class is what picks the new pool
  resources:
    requests:
      storage: 1Ti
  dataSource:
    name: vol-1-snap
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Nothing in that manifest names a bucket, because nothing has to. Both ends are resolved for you:

That last row is the one thing you have to set up yourself. A pool can only read a bucket it has been granted, so the source pool's backup bucket goes in the target pool's s3_bucket_access_names:

# pools/io2-a/terragrunt.hcl — the pool being restored INTO
s3_bucket_names        = ["mgxs3storage-io2a"]  # owned: this pool's blocks
s3_backup_bucket_names = ["mgxs3backup-io2a"]   # owned: this pool's snapshots

s3_bucket_access_names = ["mgxs3backup-gp3a"]   # gp3-a's backup bucket — the
                                                # pool io2-a may restore from

It is a list, so add one entry per pool you want to be able to restore from here. The grant is one-way: this lets io2-a pull gp3-a's snapshots, not the other way round. Terraform creates no bucket for these names — they belong to the other pool — it only adds them to this pool's IAM role.

#Create a volume

A claim and the workload that mounts it. Nothing at the pod level knows the volume is backed by object storage:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vol-1
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: mgxcsi-sc-fast
  resources:
    requests:
      storage: 1Ti
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pg
spec:
  replicas: 1
  strategy:
    type: Recreate                 # RWO: the old pod must let go first
  selector:
    matchLabels:
      app: pg
  template:
    metadata:
      labels:
        app: pg
    spec:
      securityContext:
        fsGroup: 999               # a fresh filesystem is owned by root
      containers:
        - name: postgres
          image: postgres:17
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: pg
                  key: password
            - name: PGDATA
              value: /data/pgdata  # a subdirectory, not the mount root
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: vol-1

kubectl apply -f pg.yaml
kubectl get pvc vol-1 -w                     # waits for Bound
kubectl rollout status deploy/pg
kubectl exec deploy/pg -- df -h /data

Four things in that manifest are worth knowing rather than copying:

The 1Ti is thin-provisioned — it is the size the pod sees, not capacity reserved up front in the bucket. What it does reserve is cache: the class's ratios and ceilings are applied to that figure at creation, and that slice counts against the pool whether the volume is full or empty. See Caching and tiering.

Classes ship with allowVolumeExpansion: true, so growing a volume is editing the claim's resources.requests.storage. Shrinking is not supported.

To restore into a new volume instead of creating an empty one, add a dataSource — see Snapshots and restore.

#Listing volumes and their state

Kubernetes shows you the claim. The volume behind it is the CLI, on any node:

mgx-core:127.0.0.1:main:main:admin> storage volume list

Every volume the cluster knows about, with its status, the pool and node serving it, its size, its cache slices and its bucket. --filters takes a comma-separated list of fields to return, which is how you get a table narrow enough to read:

mgx-core:127.0.0.1:main:main:admin> storage volume list --filters name,status,storage_pool,sc_node,size
mgx-core:127.0.0.1:main:main:admin> storage volume show --name vol-1

show is the whole record for one volume — including error, which is where the reason lives when a volume is stuck somewhere other than READY.

#In Grafana

Where the deployment has metrics on, the Storage dashboard is the per-volume view: what is provisioned, and each volume's throughput, IOPS, read and write latency and cache behaviour, labelled by volume name, pool and node. It is the dashboard to open when a volume is READY but slow, where storage volume show is the one for a volume that is not READY at all.

Grafana runs on the pool's VIP node in single pool mode, and on the management plane in multi pool mode, where every pool's metrics are federated into one place.

#Idle volumes

A volume nothing is using still costs something: an NVMe-oF listener, an s3backer and nbdkit process pair, and their share of the node's CPU and memory. The controller reclaims that automatically.

Its reconciler wakes every controller.timeoutVolumeCheck minutes (2 by default), lists every PV provisioned by csi.migrx.io, and compares it against the set of claims any pod in the cluster references:

• Referenced by a pod — the volume is stamped with a migrx.io/last-used annotation, and started on the backend if it was idle. A pod that lands on a volume that has been idled does not fail; it waits for the start.
• Referenced by nothing for longer than controller.idleVolumeMin minutes (10 by default) — the volume is stopped on the backend and the annotation cleared.

Stopping tears down the volume's backend processes and closes its NVMe-oF target. The volume itself is untouched — its blocks are in the bucket, its metadata is in the cluster — and the next attach starts it again and warms the cache from object storage the way a new volume does.

Two things it deliberately does not do. It does not free the volume's cache reservation: the read and write slices stay assigned to it, so idling returns CPU, memory and a listener, not pool capacity — a pool of idle volumes is still a full pool. And it does not touch S3, so nothing about idling costs a byte of traffic.

A volume provisioned well ahead of the workload that will use it gets idled before it is ever attached. That is working as intended: the first pod to claim it pays the start, and nothing was burning a node in the meantime.

#Reclaiming space in S3

Deleting files inside a volume frees space in the filesystem and nowhere else. The discard never reaches object storage during normal operation: the blocks behind those files stay allocated in the bucket, and the delete is a metadata update on the node. That is deliberate — propagating every discard would put a write path behind every rm, and the point of the cache is that it does not.

Space is reclaimed on demand instead, with the clean operation, from the CLI on any node:

mgx-core:127.0.0.1:main:main:admin> storage volume clean --name vol-1 --timeout 600 --force no

It restarts the volume, so plan it. The volume must be READY to start, and what happens then is: the NVMe-oF listener is closed so nothing can connect, the device is mounted on the node, the trim runs, the device is unmounted, and the volume is stopped for the reconciler to start again. It moves READY → CLEANING → PENDING → READY, and it is unavailable to its workload for the whole of that — this is a maintenance operation on a quiesced volume, not an online one.

Which is what --force decides. Without it the operation declines to touch a volume that has a client attached; with it, the client's I/O fails for the duration.

#Delete

Deleting the PVC deletes the volume, because the classes above use the Delete reclaim policy. Retain keeps both the PV and the backend volume when the claim goes — the mgxcsi-sc-fast tier is written that way on purpose, since an accidental kubectl delete pvc on a database should not be the last event in its life.

kubectl delete deploy pg      # the pod first, so the volume is unpublished
kubectl delete pvc vol-1

On the backend the volume moves to DELETING, its processes are torn down, and then — if the pool's storage config has storage_s3purge on, which is how the node image ships it — every object under the volume's prefix is deleted from the data bucket. The purge is idempotent and resumable: a pass that is cut short by the command timeout, or that races a writer still flushing, leaves the volume in DELETING, and the next reconcile tick purges again until the prefix is empty. Only then is it DELETED.

Two consequences worth planning around:

• A large volume takes a while to disappear. The prefix is deleted a page at a time, and a volume holding millions of 1 MiB objects is millions of deletions. The claim is gone from Kubernetes long before the bucket is.
• Snapshots are separate. They live in the backup bucket, under their own lifecycle, and deleting the volume does not delete them — see Snapshots and restore.

With storage_s3purge off, deleting the volume leaves its blocks in the bucket: recoverable if the delete was a mistake, and an orphaned prefix you are paying for if it was not.