backedblock.io

Install the K8s CSI driver

From nothing to a mounted volume: install the CSI driver against a running storage cluster, provision a PVC, mount it in a pod.

This is the Kubernetes half, and it assumes the storage cluster already exists. If it does not, start at Install the storage cluster — that page ends with the four values the Helm install below needs.

Prerequisites

  • A storage cluster reachable from the Kubernetes nodes, with the management API exposed (default port 8082) and API credentials to hand — see Install the storage cluster.
  • Kubernetes 1.20+ with the CSI feature set enabled.
  • Helm 3.

Step 1 (Optional) — Install the snapshot CRDs

Skip this if the cluster already has them.

sh
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

If the cluster has no snapshot controller of its own, let the chart deploy one with --set externalSnapshotter.enabled=true below. A single snapshot controller serves every CSI driver in a cluster, so do not add a second.

Step 2 — Install the driver

Two things go into this command: the cluster connection from Install the storage cluster, and the StorageClass parameters Sizing the pool printed — the cache ratios, the ceilings and the QoS caps, worked out from the volumes the pool has to serve.

sh
helm install mgx-csi-driver oci://docker.io/migrx/mgx-csi-driver \
  --namespace mgx-system --create-namespace \
  --version 0.1.0 \
  --set csiSecret.clusterConfig.nodes='{172.31.96.14:8082,172.31.96.15:8082,172.31.96.16:8082}' \
  --set csiSecret.clusterConfig.username=admin \
  --set csiSecret.clusterConfig.password=secret \
  --set storageclass.ratioCacheRCacheSize=0.1 \
  --set storageclass.ratioCacheRWCacheSize=0.05 \
  --set storageclass.maxCacheRCacheSize=476837 \
  --set storageclass.maxCacheRWCacheSize=238419 \
  --set storageclass.qosRWIOPS=3000 \
  --set storageclass.qosRWMBPerSec=125 \
  --set externalSnapshotter.enabled=true

nodes is every management API endpoint of the storage cluster as <ip>:8082 — list all of them, the driver fails over between them — and the credentials are its admin user.

The StorageClass parameters from the calculator

Every value in the calculator's second block, and the Helm flag it goes in as. The parameter names are what a hand-written StorageClass uses instead — see StorageClass parameters.

Sizing outputHelm valueWhat it is
ratio_cache_r_cache_sizestorageclass.ratioCacheRCacheSizeRead cache as a ratio of volume size. This is the volume's whole footprint on the cache disks: it serves reads and writes alike.
ratio_cache_rw_cache_sizestorageclass.ratioCacheRWCacheSizeHow much of that footprint may be dirty and waiting to drain — half the read ratio, so half the cache stays clean for reads.
max_cache_r_cache_sizestorageclass.maxCacheRCacheSizeRead-cache ceiling, MiB. The cache is clamp(ratio × volume size, floor, ceiling), so this is what a volume at the sized size actually gets.
max_cache_rw_cache_sizestorageclass.maxCacheRWCacheSizeWrite-cache ceiling, MiB — inside the read cache, not beside it.
qos_rw_ios_per_secstorageclass.qosRWIOPSCombined read+write IOPS cap per volume, the per-volume IOPS the pool was sized for.
qos_rw_mbytes_per_secstorageclass.qosRWMBPerSecCombined read+write bandwidth cap per volume, MB/s.

These have to match the pool they point at. The ceilings are what the pool's per-disk r_cache_size_in_mib and rw_cache_size_in_mib budgets were sized from, so a class with larger ones places fewer volumes per node than the sizing assumed, and the scheduler starts rejecting volumes earlier than expected — see Caching and tiering.

Nothing here is required: leave a parameter out and the chart's default applies, and an unset QoS cap is omitted from the class entirely rather than sent as a zero. The rest of the chart's values are on the Helm values page.

Step 3 — Verify

sh
kubectl get pods -n mgx-system
kubectl get storageclass mgxcsi-sc
kubectl get volumesnapshotclass mgxcsi-snapshotclass

You should see one mgxcsi-controller-0 pod and one mgxcsi-node-* pod per node, all Running. The chart creates a StorageClass named mgxcsi-sc and a VolumeSnapshotClass named mgxcsi-snapshotclass by default.

Step 4 — Create a test pod with a volume

A PVC and a pod that mounts it, in one file — test-pod.yaml. Nothing about the volume is special at the pod level:

yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: mgxcsi-sc
  resources:
    requests:
      storage: 1Ti
---
apiVersion: v1
kind: Pod
metadata:
  name: mgx-test
spec:
  containers:
    - name: app
      image: busybox
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data
sh
kubectl apply -f test-pod.yaml
kubectl get pvc data -w                    # waits for Bound
kubectl wait --for=condition=Ready pod/mgx-test --timeout=5m

The volume is formatted and mounted by the node plugin, so df inside the container is the check that the whole path worked:

sh
kubectl exec mgx-test -- df -h /data
text
Filesystem                Size      Used Available Use% Mounted on
/dev/nvme1n1           1006.9G     28.0K    955.6G   0% /data

A terabyte, minus the filesystem's own reserve. Then clean up — the pod first, so the volume is unpublished before it is deleted:

sh
kubectl delete pod mgx-test
kubectl delete pvc data

The class ships with reclaimPolicy: Delete, so deleting the PVC deletes the volume on the pool too — the storage group in the CLI is where to confirm it is gone. Or keep the volume and carry on with Provisioning volumes.

Next