Snapshots and restore

Snapshots are backups of a volume in object storage, taken through the standard VolumeSnapshot API and restorable as new volumes. Because the durable tier is already S3, a snapshot is a bucket-to-bucket copy rather than a read of the volume — and it can be sent to whichever S3 storage class is cheap enough for how long you intend to keep it.

#Prerequisites

The snapshot.storage.k8s.io CRDs and a snapshot controller must exist in the cluster — one controller serves every CSI driver, so use the cluster's if it has one. See Install the K8s CSI driver.

The chart creates a VolumeSnapshotClass named mgxcsi-snapshotclass.

#Full and incremental

A volume's backup is one record with a chain: one full mirror, taken the first time, and then a restore point per run after it. Every point is independently restorable, so the history is a timeline rather than a single latest copy.

 t0    t1    t2    t3
 ●─────○─────○─────○
 │     │     │     │
 ▼     ▼     ▼     ▼
full  incr  incr  incr
╔═ s3 ══════════════════╗
║ STANDARD → GLACIER_IR ║
╚═══════════════════════╝

incremental on the snapshot class is what decides this, and it is the one setting to get right before you rely on it:

#How an increment is built

The backup of a volume is a prefix in the backup bucket, laid out in two parts:

s3://mgxs3backup-gp3a/vol-1/
├── current/                  ← a mirror of the volume as it is now
└── incremental/
    ├── 20260608T143005Z/     ← the PRIOR version of every object this run
    ├── 20260609T143012Z/       overwrote or deleted
    └── 20260610T143008Z/

Every run mirrors the volume's data prefix into current/, and the objects that would be overwritten or deleted in doing so are diverted into a fresh incremental/<stamp>/ instead of being thrown away. The stamp is the run's UTC time, so the folder names sort lexically.

The consequence is worth being precise about: current/ always holds the newest data, and the increments hold progressively older versions. The chain runs backwards from now, not forwards from the full. That is what makes every point restorable without keeping a full copy per point — but it also means the points are not independent objects, and the order matters everywhere below.

Restoring a point in time therefore replays the chain in reverse:

1. Copy current/ into the target as a baseline — the newest state.
2. Overlay every increment created strictly after the target stamp, newest first. Each one holds the pre-change versions, so applying them walks the data backwards; the target's own version is written last and wins.

Restoring the latest point overlays nothing and is just current/. The sequence runs as several copy jobs across reconcile ticks, and the volume is provisioned only once the final step lands.

#Deleting restore points, oldest first

Because the chain is back-ordered, a restore point in the middle of it is load bearing — the newer increments are what get you back past it. So:

• Only the oldest restore point may be deleted. Deleting a newer one is refused with an error.
• Deleting the oldest purges its increment folder and drops it from the chain. The record and the remaining points survive.

Freeing space is therefore always the same operation repeated: delete the oldest, then the next oldest. Retention ("keep the last N") is a caller-side policy expressed exactly that way — nothing in the plugin auto-deletes. In Kubernetes that falls out naturally, since deleting old VolumeSnapshot objects oldest-first is what a retention policy does anyway.

max_increments on the snapshot config is a backstop rather than retention: once a chain holds that many points, arming a new one is refused until you delete the oldest. It deletes nothing on its own, which is what keeps the chain and the Kubernetes objects in lockstep. Each pool ships with it set to 5, so a volume snapshotted on a schedule with nothing pruning the old points starts failing at the sixth — raise it, or delete oldest-first as you go.

#Choosing where snapshots land

storageClass on the snapshot class is the destination S3 storage class: STANDARD, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING or GLACIER_IR. Each pool ships with GLACIER_IR configured, which is the cheapest class a restore can still read directly.

That default is most of the economic argument for snapshotting here rather than on the block layer. Glacier IR is $0.004 per GB-month against $0.05 for EBS snapshot storage — an order of magnitude on the same retained bytes, before counting that these are incremental too.

It is a per-class setting, so the usual arrangement is several snapshot classes — a daily one to STANDARD_IA, a monthly one to GLACIER_IR — rather than one class with a bucket lifecycle rule doing the tiering behind your back.

helm upgrade mgx-csi-driver oci://docker.io/migrx/mgx-csi-driver \
  --namespace mgx-system --reuse-values \
  --set volumeSnapshotClass.storageClass=STANDARD_IA

GLACIER and DEEP_ARCHIVE are not on the list: they need an explicit S3 restore before their objects can be read, and a restore here reads them straight out of the bucket. Every parameter is on the VolumeSnapshotClass page.

#How the copy runs

A snapshot never reads the volume. It is an S3-to-S3 copy of the volume's prefix from the pool's data bucket into its backup bucket, driven by an rclone daemon running on the node that owns the volume.

Those copies are server side wherever S3 will do them: the node issues CopyObject and the bytes move inside S3, so a snapshot costs requests and almost no node bandwidth. Where a server-side copy is not possible — a cross-bucket copy that also changes storage class is the usual case — rclone falls back to streaming the objects through the node, which works identically but does spend bandwidth and time.

That fallback is visible rather than silent: snapshot_server_side_copy_bytes sitting flat while snapshot_bytes climbs is exactly the tell, and the Server-side copy share panel on the Snapshot dashboard is the same number as a percentage.

Three settings on the snapshot config tune the concurrency, and they guard node memory and S3 request rate rather than bandwidth:

#Restoring into another pool

A restore does not have to land in the pool the snapshot came from. Give the restore claim a storageClassName whose tier resolves elsewhere and the management plane places it there, reading the source pool's backup bucket and writing the target pool's own data bucket.

That is the only way a volume moves between pools, and it needs the source pool's backup bucket in the target pool's s3_bucket_access_names — see Moving a volume to another pool for the grant and for what resolves what.

#End to end

vol-1 exists and has a workload on it. Take a full, add two increments, then restore the middle one into a new volume.

1. The first snapshot is the full. Nothing says so — the plugin creates the record and mirrors the volume into current/ because there is no record yet:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: vol-1-t0
spec:
  volumeSnapshotClassName: mgxcsi-snapshotclass
  source:
    persistentVolumeClaimName: vol-1

kubectl apply -f vol-1-t0.yaml
kubectl wait --for=jsonpath='{.status.readyToUse}'=true \
  volumesnapshot/vol-1-t0 --timeout=60m

Give it a generous timeout. The first one copies the volume's whole footprint; every one after it copies the delta.

2. Increments are the same manifest under a new name. The driver calls the same operation every time and the plugin decides full versus increment from whether the record exists, so there is nothing to switch on:

sed 's/vol-1-t0/vol-1-t1/' vol-1-t0.yaml | kubectl apply -f -
# ... the workload runs, data changes ...
sed 's/vol-1-t0/vol-1-t2/' vol-1-t0.yaml | kubectl apply -f -

kubectl get volumesnapshot

NAME       READYTOUSE   SOURCEPVC   RESTORESIZE   SNAPSHOTCLASS          AGE
vol-1-t0   true         vol-1       1Ti           mgxcsi-snapshotclass   3h
vol-1-t1   true         vol-1       1Ti           mgxcsi-snapshotclass   2h
vol-1-t2   true         vol-1       1Ti           mgxcsi-snapshotclass   1h

Three Kubernetes objects, one backup record with three restore points in it.

3. Restore the middle point into a new volume. A restore is a PVC whose dataSource names the snapshot for the point you want:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vol-1-t1-restored
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: mgxcsi-sc-standard
  resources:
    requests:
      storage: 1Ti
  dataSource:
    name: vol-1-t1
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

kubectl apply -f restore.yaml
kubectl get pvc vol-1-t1-restored -w        # waits for Bound

Behind that claim: current/ is copied in as the baseline, then the t2 increment is overlaid on top of it — every increment newer than t1 — which walks the data back to how it stood at t1. Then the volume is provisioned and the claim binds.

The original is untouched. A restore is always a new volume, never an in-place rollback, and it starts cold: its blocks are in the bucket and its cache is empty, so it warms as it is read.

#Checking a snapshot

From Kubernetes, the claim-level view — is it usable, how big, and what went wrong if it is not:

kubectl get volumesnapshot
kubectl describe volumesnapshot vol-1-t1
kubectl get volumesnapshotcontent \
  -o custom-columns=NAME:.metadata.name,HANDLE:.status.snapshotHandle

That handle is <record>@<stamp> — the backup record (named after the volume) and the restore point inside it. It is the string that ties a VolumeSnapshot to what the CLI shows.

From the CLI, the backup itself, on any node:

mgx-core:127.0.0.1:main:main:admin> snapshot snapshot list
mgx-core:127.0.0.1:main:main:admin> snapshot snapshot show --name vol-1
mgx-core:127.0.0.1:main:main:admin> snapshot config show --name <pool>

--filters takes a comma-separated list of fields to return, which is how you get a table narrow enough to read:

mgx-core:127.0.0.1:main:main:admin> snapshot snapshot list --filters name,kind,volume,status,stamp

In Grafana, the Snapshot dashboard, filtered by pool and by snapshot: how many exist and how many are READY, total logical size, transfer errors and jobs with fatal or retryable errors, average elapsed versus transfer time, speed, and the server-side copy share. It is the view for whether backups are keeping up; snapshot snapshot show is the one for why a single backup did not.

Grafana runs on the pool's VIP node in single pool mode and on the management plane in multi pool mode.

#Deleting snapshots

deletionPolicy on the snapshot class decides what happens to the backup in S3 when the VolumeSnapshot object goes away — Delete (the default) removes the restore point, Retain leaves it in the bucket and untracked by Kubernetes.

Deletion is oldest-first, for the reason in Deleting restore points: delete vol-1-t0 before vol-1-t1. Deleting the newest point of a chain while older ones remain is refused, and the VolumeSnapshot will not go away until it succeeds.