Multi pool mode

A management plane and N pools, driven by Terragrunt over the same modules. State is in S3 per unit, provisioning is agentless over SSM, and each pool is a folder with a ~25-line terragrunt.hcl — so adding the tenth pool costs the same as adding the second.

Units are applied in dependency order — network → pools → mgmt — and Terragrunt works that out for itself.

#Step 1 — Clone and bootstrap the state bucket

Every unit but this one keeps its state in S3, so the bucket comes first. Set state_bucket and region in bootstrap/terraform.tfvars to match common.hcl, then:

git clone https://github.com/migrx-io/backedblock.git
cd backedblock/terragrunt-scale
(cd bootstrap && terraform init && terraform apply)

bootstrap keeps local state; every other unit uses the bucket it just made.

#Step 2 — Size the pools

Volumes first, machines second. Sizing the pool turns the volumes one pool has to serve into the block for Step 4 and the StorageClass parameters for the driver install afterwards.

If the workload needs more nodes than one pool should carry, the calculator says so and gives the split — which is the layout this page is for: several pools of the same shape rather than one oversized one.

#Step 3 — Store the secrets in SSM

cp secrets.env.example secrets.env       # then fill in real values
aws ssm put-parameter --type SecureString \
  --name /mgx/main/secrets --value file://secrets.env

Every line is KEY=value and every one has to be filled — the placeholders are not defaults, and a node that reads an unset value will come up unusable.

What each secret is

Variable	What it is
`CASS_USER` · `CASS_PASSWD`	The Cassandra superuser the nodes create and then authenticate with. Internal to the cluster — nothing outside it uses these.
`MGX_GW_X_API_KEY`	The API key the management API accepts in the `X-API-KEY` header.
`MGX_X_API_KEY`	The same idea for a node's own API. It does not have to differ from the gateway key, which is why the example carries one placeholder for both.
`MGX_GW_ADMIN_PASSWD`	Password for the `admin` account on the API. This is the one you need again later — it is what the CSI driver authenticates with, and what the CLI logs in with.
`GRAFANA_USER` · `GRAFANA_PASSWD`	Grafana's admin login, used only when `enable_metrics` and `enable_grafana` are on.

#Step 4 — Configure

Three files, in order of how often they change: common.hcl for everything shared, pools/_pool.hcl for the pool shape, and one terragrunt.hcl per pool for what makes that pool itself.

common.hcl — the single source of truth

Value	What to set
`state_bucket` · `cluster` · `region`	From Step 1. `cluster` names the cluster every node joins.
`vpc_id`	The existing VPC to deploy into.
`azs`	The AZs the network builds subnets in — one mgmt and one storage subnet each.
`mgmt_subnet_cidrs` · `storage_subnet_cidrs`	One free CIDR per AZ, in the same order as `azs`.
`bastion`	`vpc_subnet` is an existing public subnet, which also hosts the NAT gateway; `ami` an Ubuntu image for your region; `whitelist_ips` ships as `0.0.0.0/0` — narrow it.
`nodes_ami`	The published node image for your region — see Node AMIs.
`key_name` · `ssh_public_key_path`	The key pair for SSH access to the nodes. Provisioning itself is agentless over SSM.

azs is a network-level list. Each pool pins itself to one of them with its own az, because EBS cache volumes are AZ-bound — so pools can be spread across zones, but a pool does not straddle them.

pools/_pool.hcl — the shape every pool shares

This is where the output of Step 2 goes, once, for every pool that shares a shape: nodes_count, nodes_instance_type, nvme_node_disks_count, max_volumes_count, the two cache sizes and the ebs_volumes block — see Caching and tiering.

Two settings differ from a standalone pool, and both are right for this layout:

Value	Why
`enable_grafana = false`	Grafana runs on the management plane instead, where it can see every pool.
`cross_peer_scrape = false`	Each node scrapes only itself and `mgmt` federates them all, so there is no node-selection single point of failure.

A pool that needs a different shape — bigger volumes, more throughput — carries its own overrides in its terragrunt.hcl rather than changing this file.

pools/<pool>/terragrunt.hcl — one per pool

Value	What to set
`pool_name` · `description`	Name of the pool and the prefix on its resources.
`az`	The single AZ this pool sits in, one of the network's `azs`.
`labels`	Comma-separated `key=value` pairs. These are what a scheduler config's `pool_selector` matches, so they are how volumes get steered to one pool rather than another — see Volume scheduling.
`s3_bucket_names` · `s3_backup_bucket_names`	Globally unique bucket names — block data, and snapshots.

Everything else comes from _pool.hcl and common.hcl.

#Step 5 — Apply

Terragrunt builds the dependency graph and applies network → pools → mgmt:

terragrunt run --all -- apply

Why the order matters

mgmt has to come after the pools, and has to be re-applied whenever you add or remove one. Each pool's apply publishes its node addresses to SSM (/mgx/<cluster>/pools/<pool>); mgmt's apply reads that path back to build its registry, so the entries must already exist. run --all handles the ordering.

#Step 6 — Wait for provisioning

In SSM mode an apply only creates the provisioning resource — setup-node.sh then runs asynchronously on each node, so the command returns before the nodes are ready. Watch the associations:

aws ssm list-associations --region us-east-1 \
  --query "Associations[?starts_with(AssociationName, \`mgx-\`)].[AssociationName,Overview.Status]" \
  --output table

Everything Success means provisioning finished.

#Step 7 — Connect to the cluster

mgx-cli is on PATH on every node, and the management nodes see every pool:

(cd network && terragrunt output bastion_public_ip)
(cd mgmt    && terragrunt output node_private_ips)

ssh -J ubuntu@<bastion_public_ip> ubuntu@<mgmt_node_ip>
mgx-cli

text

mgx-core:127.0.0.1:nologin> login admin --cluster main --ns main
Password:
 logged..

The password is the MGX_GW_ADMIN_PASSWD from Step 3. The cluster, with the node currently holding the VIP:

text

mgx-core:127.0.0.1:main:main:admin> cluster list

Every pool, each of which has to read READY:

text

mgx-core:127.0.0.1:main:main:admin> cache pool list

A pool missing from that list is a pool mgmt has not discovered — re-apply mgmt and look again.

#Step 8 (Optional) — Grafana

Metrics are on by default here, and Grafana runs on the management plane rather than in a pool, because that is where every pool's metrics are federated:

hcl

enable_metrics = true      # mgmt/terragrunt.hcl
enable_grafana = true

It has no public address, so forward the port over the bastion:

ssh -N -J ubuntu@<bastion_public_ip> -L 3000:localhost:3000 ubuntu@<mgmt_vip_node_ip>

and open http://localhost:3000, with the GRAFANA_USER and GRAFANA_PASSWD from Step 3. Two dashboards matter:

Dashboard	What it is for
Cache	The state of the cache nodes, across every pool — one row per node, and on a healthy fleet every one of them is green.
Storage	The volumes themselves — what is provisioned, and each volume's metrics.

#Step 9 — Point the driver at the management plane

The CSI driver points at the mgmt nodes, which present one API for every pool beneath them:

(cd mgmt && terragrunt output node_private_ips)

Each of those on port 8082 is an API endpoint for csiSecret.clusterConfig.nodes — carry them into Install the K8s CSI driver, with admin and MGX_GW_ADMIN_PASSWD as the credentials.

#Day-2

Task	Action
Add a pool	`scripts/new-pool.sh pool3`, `(cd pools/pool3 && terragrunt apply)`, then re-apply `mgmt`. Nothing in `mgmt` to edit — it discovers pools itself.
Remove a pool	`terragrunt destroy` in its folder, delete the folder, re-apply `mgmt`.
Scale a pool	Change `nodes_count`, apply that pool.
Maintain a node	Drain it first — see Pool node management.
Tear down all	`terragrunt run --all -- destroy`.