Troubleshooting

A PVC that stays Pending

The claim's events

The claim's events are the provisioning log, and on this driver they are more informative than they first look — the pool's own volume state is quoted in them, so describe on the claim tells you what the storage side is doing without leaving kubectl:

sh
kubectl describe pvc data
text
Events:
  Type     Reason                 Age   Message
  ----     ------                 ----  -------
  Warning  ProvisioningFailed     66s   rpc error: code = Aborted desc = volume vol-12a6ea389ccdf9e8 is creating
  Warning  ProvisioningFailed     51s   rpc error: code = Aborted desc = volume vol-12a6ea389ccdf9e8 is not READY: INIT
  Warning  ProvisioningFailed     35s   rpc error: code = Aborted desc = volume vol-12a6ea389ccdf9e8 is not READY: PENDING
  Normal   Provisioning           5s    External provisioner is provisioning volume for claim "default/data"
  Normal   ProvisioningSucceeded  5s    Successfully provisioned volume pvc-12477922-8afa-4c6e-b8e4-099c68462f5a

That is a success. Every Warning ProvisioningFailed in it is expected. Creating a volume is asynchronous on the pool, so the driver answers the CreateVolume call with Aborted and the volume's current status for as long as it is not READY; Aborted is retryable, so the external provisioner backs off and asks again, and each attempt leaves another warning. Read the Age column and the sequence is the pool's state machine: creating → INIT → PENDING → provisioned, about a minute for the volume above. A claim that binds after a run of warnings has not recovered from anything — that is the normal path, and it is why a bare warning count is not a signal.

What is a signal is the message and whether the status advances:

What you seeWhat it means
is creating / is not READY: INIT / is not READY: PENDING, advancingWorking. Keep waiting.
is not READY: INIT repeating and never reaching PENDINGThe scheduler never placed the volume — see A volume that will not leave INIT below.
is not READY: PENDING repeatingPlaced, but the node cannot bring it up. storage volume show and its error field are the next stop.
Anything that is not a not-READY status — rejected credentials, an unreachable endpoint, a parameter the pool refusesA real error from the management API, quoted verbatim after desc =.
ExternalProvisioning only, with no events from csi.migrx.io at allNothing is answering for the driver. Check mgxcsi-controller-0 is Running, then kubectl -n mgx-system logs mgxcsi-controller-0 --all-containers --tail=50.

Once the claim is Bound, provisioning is over and anything still wrong is attach or mount, which shows up on the pod instead:

sh
kubectl describe pod mgx-test

The volume's state on the pool

Those messages carry the volume's name on the pool — vol-12a6ea389ccdf9e8 above. That is the handle the CLI wants, and it is in the events from the first attempt, so it is available while the claim is still Pending and has no PV. It is not the PV name (pvc-<uid>), which only exists after binding.

The event tells you the status. The volume's own record tells you why it is in it — which pool and node it landed on, its cache slices, and its error field. From any node:

text
mgx-core:127.0.0.1:main:main:admin> storage volume show --name vol-12a6ea389ccdf9e8
mgx-core:127.0.0.1:main:main:admin> storage volume list --filters name,status,storage_pool,sc_node,error

show is the whole record for the one volume, error included; list with --filters is the view to take when you want the volume in the context of everything else the pool is serving — a volume stuck in INIT next to a pool with no free node is a different diagnosis from one stuck on its own. The full status list is in Provisioning volumes.

A volume that will not leave INIT

INIT means created but not placed: the scheduler has not found a pool, node and disk with room for it, and it keeps trying. Almost always the pool is full in one of the three ways it can be full — and none of them is S3 capacity, which is why a pool with an almost empty bucket still refuses volumes:

GuardWhat to check
Volume countThe volumes on each node against the pool's max_volumes_count. Count them from storage volume list --filters name,status,sc_node — the cap is per node, and the scheduler needs a node that stays under it with the new volume.
Read cacheThe read slices already committed on a node against r_cache_size_in_mib × nvme_node_disks_count.
Write cacheThe same against rw_cache_size_in_mib × nvme_node_disks_count.

The pool side of those numbers is the cache group — cache pool list for the pools and their state, help cache for the config and node-mount commands:

text
mgx-core:127.0.0.1:main:main:admin> cache pool list
mgx-core:127.0.0.1:main:main:admin> cluster list

Then compare them with what the new volume is asking for. A volume's slices are clamp(ratio × volume size, floor, ceiling) from its StorageClass, so a large volume asks for the class's ceilings — a class with a 20 GiB read ceiling needs 20 GiB free on one node's cache, not 20 GiB free across the pool. That arithmetic, and what to change when it does not fit, is Caching and tiering; the guards themselves are Volume scheduling.

Three other reasons a candidate is out of the running, worth ruling out before resizing anything: a node that is not READY (cluster list, and the Cache tab of bbctl ui, where an unhealthy node shows down), a node that is draining, and a scheduler config whose selectors match no pool — see Selectors.

A new volume is slower than its QoS caps

Expected, for a while, and it is the filesystem rather than the storage. The node plugin formats a new volume before the first mount, and a full format of a multi-terabyte device would take far too long to do inline — so the metadata write is left to finish lazily in the background after the volume is already mounted and serving. On a large volume that background pass runs for 30 minutes to an hour, and while it does it is spending the volume's own IOPS and bandwidth on itself. A benchmark started immediately after the PVC binds is measuring the workload plus the initialisation, so it comes in under the caps the StorageClass sets.

Nothing to fix. The volume reaches the performance its class promises once the pass is done, and the way to confirm it is the Volumes tab of bbctl ui, with the volume ticked: a freshly created volume with no workload on it still shows write IOPS and throughput while the initialisation is running, and they fall to nothing when it finishes. Benchmark after that, not before — and if a volume is being created specifically to benchmark, create it early and let it settle.

Latency spikes

Two different causes, one per direction, and the fix for both is cache size.

Reads. A read whose block is not in the read cache fetches the 1 MiB block containing it from object storage, so it pays an S3 round trip instead of a local disk read. A volume that spikes on reads is a volume whose working set does not fit its read cache, or one that is still cold — freshly created, freshly restored, or freshly started after being idled, all of which begin with an empty cache and warm up from the bucket. Sequential access hides most of this, because one 1 MiB block satisfies many consecutive reads; random access over a working set larger than the cache does not. What a miss actually costs is in Tradeoffs.

Writes. A write is acknowledged into the write cache and drained to object storage in the background. Sustained writes that outrun the drain rate fill the dirty-data allowance, and once it is full the volume can only accept writes as fast as it drains them — the write path throttles, and that shows up as a latency spike rather than a gradual slope. Bursts are what the cache is for; this is what happens when the burst does not end.

Both are read off the Volumes tab of bbctl ui, which carries per-volume read and write latency alongside the cache sizes and cache behaviour — a write-cache figure sitting at its limit while latency climbs is the throttle, and read latency climbing with a cold or churning cache is the miss path. See Install the storage cluster.

The lever is the volume's cache slices — ratioCacheRCacheSize and maxCacheRCacheSize for the read side, ratioCacheRWCacheSize and maxCacheRWCacheSize for the dirty-data allowance inside it. Raising them costs placement density: every volume reserves its slice on a node's cache whether it uses it or not, so a class with bigger slices fits fewer volumes per node and the pool starts refusing volumes sooner. The sizes are computed once, when the volume is created, so editing a class does not resize the volumes already provisioned — a new class and a new volume are what a change applies to. See Caching and tiering.