← all writeups
One bad limit and the whole node went with it
A single memory limit bump turned a routine deploy into a rolling cascade across three nodes.
- Date
- 2026-03-02
- Type
- postmortem
- Severity
- SEV2
- Duration
- 1h 48m
- Impact
- Elevated latency across two services. No customer-facing errors.
PLACEHOLDER POST — replace with your real writeup.
Someone raised a memory limit to stop an OOMKill. It worked, in that the pod stopped being killed. Everything else got worse.
Why it cascaded
The higher limit meant fewer pods fit per node. The scheduler packed the remaining pods tighter, which pushed a second workload over its limit, which evicted it, which rescheduled it onto an already-tight node.
resources:
requests:
memory: 512Mi # unchanged — this was the real bug
limits:
memory: 4Gi # raised from 1GiRequests were never updated, so the scheduler’s idea of the node’s free memory was fiction.
What changed
- Limit changes now require a matching request change in review
- A
LimitRangeblocks a limit:request ratio above 2:1 - Capacity alert fires on requested memory, not used
The fix was one line. Finding it was the whole afternoon.
← all writeups