The consumer lag that fed itself
A slow downstream made consumers miss their poll interval. Each rebalance made them slower, and lag grew to 14 million messages.
- Date
- 2025-02-11
- Type
- postmortem
- Severity
- SEV2
- Duration
- 6h 40m
- Impact
- Order notifications delayed by up to 3 hours. No messages lost.
PLACEHOLDER POST — seed data. Replace with your real writeup.
The notification service consumed order events, called a templating service, and sent email. When the templating service slowed from 40ms to 900ms, the consumer group started rebalancing. It kept rebalancing for six hours.
Timeline
| Time | Event |
|---|---|
| 08:15 | Templating service latency rises after its own deploy |
| 08:30 | First consumer exceeds max.poll.interval.ms, rebalance |
| 09:00 | Rebalances every 2–3 minutes; lag at 2M |
| 11:40 | Lag at 14M; scaled consumers, which made rebalances worse |
| 13:10 | max.poll.records cut from 500 to 50, templating rolled back |
| 14:55 | Lag cleared, incident closed |
The actual cause
Each poll returned 500 records. At 900ms apiece, a batch took longer than the five-minute poll interval, so the broker decided the consumer was dead. The rebalance revoked its partitions, and uncommitted work was redone by the next consumer, which was just as slow. Adding consumers only made the rebalances happen more often.
What changed
- Poll batch size derived from measured p99 processing time, not left at the default
- Cooperative sticky assignor, so a rebalance no longer stops the whole group
- Alert on rebalance rate, not only on lag
- Templating calls moved behind a local cache with a timeout
Scaling out a consumer group that is stuck rebalancing is pouring water on a grease fire.
← all writeups