aman®
← all writeups

The autoscaler scaled us down at peak

metrics-server went away, the HPA read that as no load, and checkout dropped to its minimum replicas in the middle of a Saturday sale.

Date
2026-01-17
Type
incident
Severity
SEV1
Duration
38m
Impact
Checkout p99 latency above 9s; ~12% of orders failed for 25 minutes.

PLACEHOLDER POST — seed data. Replace with your real writeup.

Traffic was 4x normal. Checkout was running 40 pods. Then, over about three minutes, it was running 6.

Timeline

Time Event
11:20 metrics-server pod evicted during node drain
11:23 HPA begins scaling checkout down toward minReplicas
11:27 Latency alert fires; error rate climbing
11:35 Manual kubectl scale --replicas=40 to stop the bleeding
11:41 metrics-server rescheduled, HPA recovers
11:58 Latency at baseline, incident closed

The actual cause

We were running a custom metrics adapter that returned 0 instead of an error when its upstream was unavailable. The HPA did exactly what it was told. With CPU usage reported as zero, the correct replica count was the minimum.

What changed

  • The adapter returns an error when it has no data, so the HPA holds its current scale
  • behavior.scaleDown.stabilizationWindowSeconds: 600 and a max 10% scale-down per minute
  • metrics-server runs two replicas with a PodDisruptionBudget
  • minReplicas for checkout during sale events is now set by the event calendar

An autoscaler that can’t see should hold still, not guess.

← all writeups