Troubleshooting
A PVC that stays Pending
The claim's events
The claim's events are the provisioning log, and on this driver they are more
informative than they first look — the pool's own volume state is quoted in
them, so describe on the claim tells you what the storage side is doing
without leaving kubectl:
kubectl describe pvc data
Events: Type Reason Age Message ---- ------ ---- ------- Warning ProvisioningFailed 66s rpc error: code = Aborted desc = volume vol-12a6ea389ccdf9e8 is creating Warning ProvisioningFailed 51s rpc error: code = Aborted desc = volume vol-12a6ea389ccdf9e8 is not READY: INIT Warning ProvisioningFailed 35s rpc error: code = Aborted desc = volume vol-12a6ea389ccdf9e8 is not READY: PENDING Normal Provisioning 5s External provisioner is provisioning volume for claim "default/data" Normal ProvisioningSucceeded 5s Successfully provisioned volume pvc-12477922-8afa-4c6e-b8e4-099c68462f5a
That is a success. Every Warning ProvisioningFailed in it is expected.
Creating a volume is asynchronous on the pool, so the driver answers the
CreateVolume call with Aborted and the volume's current status for as long
as it is not READY; Aborted is retryable, so the external provisioner backs
off and asks again, and each attempt leaves another warning. Read the Age
column and the sequence is the pool's state machine: creating → INIT →
PENDING → provisioned, about a minute for the volume above. A claim that
binds after a run of warnings has not recovered from anything — that is the
normal path, and it is why a bare warning count is not a signal.
What is a signal is the message and whether the status advances:
| What you see | What it means |
|---|---|
is creating / is not READY: INIT / is not READY: PENDING, advancing | Working. Keep waiting. |
is not READY: INIT repeating and never reaching PENDING | The scheduler never placed the volume — see A volume that will not leave INIT below. |
is not READY: PENDING repeating | Placed, but the node cannot bring it up. storage volume show and its error field are the next stop. |
Anything that is not a not-READY status — rejected credentials, an unreachable endpoint, a parameter the pool refuses | A real error from the management API, quoted verbatim after desc =. |
ExternalProvisioning only, with no events from csi.migrx.io at all | Nothing is answering for the driver. Check mgxcsi-controller-0 is Running, then kubectl -n mgx-system logs mgxcsi-controller-0 --all-containers --tail=50. |
Once the claim is Bound, provisioning is over and anything still wrong is
attach or mount, which shows up on the pod instead:
kubectl describe pod mgx-test
The volume's state on the pool
Those messages carry the volume's name on the pool — vol-12a6ea389ccdf9e8
above. That is the handle the CLI wants, and it is in the events
from the first attempt, so it is available while the claim is still Pending
and has no PV. It is not the PV name (pvc-<uid>), which only exists after
binding.
The event tells you the status. The volume's own record tells you why it is in
it — which pool and node it landed on, its cache slices, and its error field.
From any node:
mgx-core:127.0.0.1:main:main:admin> storage volume show --name vol-12a6ea389ccdf9e8 mgx-core:127.0.0.1:main:main:admin> storage volume list --filters name,status,storage_pool,sc_node,error
show is the whole record for the one volume, error included; list with
--filters is the view to take when you want the volume in the context of
everything else the pool is serving — a volume stuck in INIT next to a pool
with no free node is a different diagnosis from one stuck on its own. The full
status list is in
Provisioning volumes.
A volume that will not leave INIT
INIT means created but not placed: the scheduler has not found a pool, node
and disk with room for it, and it keeps trying. Almost always the pool is full
in one of the three ways it can be full — and none of them is S3 capacity,
which is why a pool with an almost empty bucket still refuses volumes:
| Guard | What to check |
|---|---|
| Volume count | The volumes on each node against the pool's max_volumes_count. Count them from storage volume list --filters name,status,sc_node — the cap is per node, and the scheduler needs a node that stays under it with the new volume. |
| Read cache | The read slices already committed on a node against r_cache_size_in_mib × nvme_node_disks_count. |
| Write cache | The same against rw_cache_size_in_mib × nvme_node_disks_count. |
The pool side of those numbers is the cache group — cache pool list for the
pools and their state, help cache for the config and node-mount commands:
mgx-core:127.0.0.1:main:main:admin> cache pool list mgx-core:127.0.0.1:main:main:admin> cluster list
Then compare them with what the new volume is asking for. A volume's slices are
clamp(ratio × volume size, floor, ceiling) from its StorageClass, so a large
volume asks for the class's ceilings — a class with a 20 GiB read ceiling
needs 20 GiB free on one node's cache, not 20 GiB free across the pool. That
arithmetic, and what to change when it does not fit, is
Caching and tiering; the guards
themselves are Volume scheduling.
Three other reasons a candidate is out of the running, worth ruling out before
resizing anything: a node that is not READY (cluster list, and the
Cache tab of bbctl ui, where an unhealthy node shows down), a node
that is draining, and a scheduler config whose selectors match no pool — see
Selectors.
A new volume is slower than its QoS caps
Expected, for a while, and it is the filesystem rather than the storage. The node plugin formats a new volume before the first mount, and a full format of a multi-terabyte device would take far too long to do inline — so the metadata write is left to finish lazily in the background after the volume is already mounted and serving. On a large volume that background pass runs for 30 minutes to an hour, and while it does it is spending the volume's own IOPS and bandwidth on itself. A benchmark started immediately after the PVC binds is measuring the workload plus the initialisation, so it comes in under the caps the StorageClass sets.
Nothing to fix. The volume reaches the performance its class promises once the
pass is done, and the way to confirm it is the Volumes tab of bbctl ui,
with the volume ticked: a freshly created volume with no workload on it still shows write
IOPS and throughput while the initialisation is running, and they fall to
nothing when it finishes. Benchmark after that, not before — and if a volume is
being created specifically to benchmark, create it early and let it settle.
Latency spikes
Two different causes, one per direction, and the fix for both is cache size.
Reads. A read whose block is not in the read cache fetches the 1 MiB block containing it from object storage, so it pays an S3 round trip instead of a local disk read. A volume that spikes on reads is a volume whose working set does not fit its read cache, or one that is still cold — freshly created, freshly restored, or freshly started after being idled, all of which begin with an empty cache and warm up from the bucket. Sequential access hides most of this, because one 1 MiB block satisfies many consecutive reads; random access over a working set larger than the cache does not. What a miss actually costs is in Tradeoffs.
Writes. A write is acknowledged into the write cache and drained to object storage in the background. Sustained writes that outrun the drain rate fill the dirty-data allowance, and once it is full the volume can only accept writes as fast as it drains them — the write path throttles, and that shows up as a latency spike rather than a gradual slope. Bursts are what the cache is for; this is what happens when the burst does not end.
Both are read off the Volumes tab of bbctl ui, which carries per-volume read and write
latency alongside the cache sizes and cache behaviour — a write-cache figure
sitting at its limit while latency climbs is the throttle, and read latency
climbing with a cold or churning cache is the miss path. See
Install the storage cluster.
The lever is the volume's cache slices — ratioCacheRCacheSize and
maxCacheRCacheSize for the read side, ratioCacheRWCacheSize and
maxCacheRWCacheSize for the dirty-data allowance inside it. Raising them
costs placement density: every volume reserves its slice on a node's cache
whether it uses it or not, so a class with bigger slices fits fewer volumes per
node and the pool starts refusing volumes sooner. The sizes are computed once,
when the volume is created, so editing a class does not resize the volumes
already provisioned — a new class and a new volume are what a change applies
to. See Caching and tiering.