Volume governance, encryption posture, quota strategy.

IAM and quotas¶

Role	Can do
volume.viewer	List volumes, view metrics
volume.builder	Create/attach/detach/delete volumes in a project
volume.admin	Above + encryption key rotation, quota changes

Default per-project quotas: 50 TiB total, 256 volumes, 100 attachments per VM. All bumpable via support ticket (≤ 1 BWD typical).

Encryption¶

All NVMe HCI volumes are encrypted at rest by default (LUKS, AES-256-XTS) with platform-managed keys. To use customer-managed keys instead:

bash cd storage volume create \ --name db-data \ --size-gib 500 \ --kms-key projects/acme-prod/keys/vol-master

CMK volumes can't be shared cross-project unless the receiving project has Reader on the key. Plan key sharing before you provision.

Tagging policy¶

Tag every volume with app, env, cost-center. The Cost Explorer rolls up by tag — without tags, you can't answer "why did storage spend jump 30% MoM?"

Project policies can require tags at create time:

bash cd project policy set acme-prod \ --require-tags 'app,env,cost-center' \ --scope volume

Snapshot policy¶

NVMe HCI volumes inherit project snapshot policy (see Snapshots & Images — Administration). The volume can override:

bash cd storage volume tag db-data \ --set snapshot-policy=critical-6h

Related¶

• Block Storage (NVMe HCI) — Operation
• Block Storage (NVMe HCI) — Troubleshooting
• Block Storage (PIOPS)

Day-2: monitoring IOPS/latency, hot-grow, snapshot hygiene, and migrating to PIOPS when needed.

Metrics¶

Metric	Healthy	Alert
disk.read_iops p95	within flavor envelope	sustained > 90% of envelope
disk.write_iops p95	same	same
disk.read_latency_ms p99	< 5 ms	> 15 ms for 10 min
disk.write_latency_ms p99	< 8 ms	> 20 ms for 10 min
disk.queue_depth	< 16	> 32 sustained

disk.read_latency_ms >15 ms p99 on HCI is a noisy-neighbour signal. Open a ticket or migrate to PIOPS.

Hot-grow¶

Online expansion is supported — no detach, no VM reboot:

```bash cd storage volume resize --volume db-data --size-gib 1000

Inside the guest:¶

sudo growpart /dev/vdb 1 sudo resize2fs /dev/vdb1 # or xfs_growfs / for XFS ```

You cannot shrink a volume online. Create a smaller volume and rsync.

Snapshot lifecycle¶

Snapshots are RoW — initial cost is metadata only, billed size grows with divergence. For a typical OLTP volume, plan for 30–60% of source size for a 7-day-old snapshot.

A quarterly snapshot audit catches orphans:

bash cd storage snapshot list --orphaned --older-than 30d

Orphans are snapshots whose source volume was deleted but the snapshot retention pinned them. Decide: keep (for forensic) or delete (saves BDT).

Migrating HCI → PIOPS¶

When workload outgrows HCI's best-effort IOPS:

```bash cd storage volume migrate \ --volume db-data \ --to-class piops-3000

Live migration; brief IO pause (~200 ms)¶

```

Migration takes 10–60 minutes depending on data volume. Billing switches at completion.

Related¶

• Block Storage (NVMe HCI) — Administration
• Block Storage (NVMe HCI) — Troubleshooting

Slow IOPS during business hours¶

HCI is best-effort — neighbour workloads share the underlying NVMe pool. Symptoms:

• disk.read_iops floors at ~5,000–8,000 during peak
• disk.read_latency_ms p99 spikes >20 ms

Resolutions, in order:

1. Confirm it's HCI (not PIOPS) — cd storage volume show --volume <id>
2. Open a Support ticket with metric screenshots — SRE may rebalance the underlying pool
3. Migrate to PIOPS if sustained — guaranteed IOPS

VolumeAttachmentFailed¶

ERROR: VolumeAttachmentFailed: volume already attached to vm 'web-02'

A volume attaches to one VM at a time. Detach first:

bash cd storage volume detach --volume vol-abc cd storage volume attach --volume vol-abc --vm web-03

If detach hangs: the source VM's guest may have the volume mounted. SSH in, umount, then detach.

resize2fs reports old size after volume resize¶

You ran cd storage volume resize but df -h still shows the old size:

bash sudo lsblk # Confirm kernel sees new size sudo growpart /dev/vdb 1 # Resize partition first sudo resize2fs /dev/vdb1 # Then the filesystem

If lsblk shows old size, the kernel hasn't rescanned:

bash echo 1 | sudo tee /sys/class/block/vdb/device/rescan

Volume stuck Detaching¶

Detach >2 min usually means in-guest mount. Force-detach via console Storage → Volume → Force detach — this may corrupt the filesystem. Snapshot before forcing.

Snapshot billed-size keeps growing¶

A snapshot of a chatty volume diverges fast. If billed-size > source-volume-size, you've kept too many snapshots too long:

```bash cd storage snapshot list --volume --output table

Prune: cd storage snapshot delete --snapshot ¶

```

CMK volume can't be read after key rotation¶

Customer-managed key rotation requires re-wrapping existing volumes; old volumes can't decrypt under the new key alone. Use the key's previous version to read, or schedule a re-encryption window.

Related¶

• Block Storage (NVMe HCI) — Administration
• Block Storage (NVMe HCI) — Operation