aman®
← all writeups

The etcd volume that quietly filled up

No compaction, no defrag, no alert on database size. The control plane went read-only in the middle of a deploy.

Date
2025-09-21
Type
postmortem
Severity
SEV1
Duration
53m
Impact
Cluster API read-only for 53 minutes; no deploys or scaling. Running workloads unaffected.

PLACEHOLDER POST — seed data. Replace with your real writeup.

Halfway through a routine rollout, kubectl apply started returning etcdserver: mvcc: database space exceeded. The API server could still read, but it could not write anything. That included the ReplicaSet the rollout had just scaled to zero.

Timeline

Time Event
14:02 Deploy begins
14:06 API writes fail with database space exceeded
14:11 etcd raises NOSPACE alarm on all three members
14:30 Compacted to current revision, defragmented one member at a time
14:48 Alarm disarmed, writes resume
14:55 Rollout completed, incident closed

The actual cause

A controller was writing a status update to a custom resource every two seconds, and each write created a new revision. Auto-compaction had been disabled in a cluster template two years earlier. The database grew until it hit the 2 GB default quota.

etcdctl endpoint status -w table            # DB SIZE 2.1 GB
etcdctl compact $(etcdctl endpoint status -w json | jq '.[0].Status.header.revision')
etcdctl defrag --endpoints=$ONE_MEMBER_AT_A_TIME
etcdctl alarm disarm

What changed

  • --auto-compaction-retention=1h restored, and the template it came from fixed
  • Alert when etcd_mvcc_db_total_size_in_bytes passes 60% of the quota
  • The chatty controller now only writes status when something has actually changed
← all writeups