aman
← all writeups

Autoscaling can't catch a burst

KEDA HTTP scaling reacted only after a burst of 10-second report calls had already failed with 502s. A warm pool fixed it, and worker threads cut the pool from 10 pods to 3.

Date
2026-06-13
Type
note
Impact
Load test only: 9 of 15 burst calls failed with 502 under reactive scaling. With a warm pool, zero failed. This is why the heavy report path doesn't rely on reactive scaling.

Our report service does about 10 seconds of synchronous CPU work per call. The real traffic pattern is ten people clicking “run report” at about the same moment. We wanted to know whether KEDA’s HTTP add-on could scale fast enough, so we built a fake: a Node endpoint that blocks the event loop for about 10s, run on EKS behind KEDA 2.20.1 and the HTTP add-on 0.14.1.

What actually happens

Test: 15 calls, 10 concurrent, starting from idle.

Config 502s median
min=1, topology DoNotSchedule 9/15 33.5s
min=1, ScheduleAnyway 8/15 22.8s
min=10 warm pool 0/15 12.5s

With DoNotSchedule, new pods were Ready after about 65s. The burst was over by then. With ScheduleAnyway, the HPA flapped between 1 and 4 pods and never built capacity.

Why

The interceptor forwards every request to existing pods immediately. targetPendingRequests is only a scaling signal, not a limit on requests per pod. The interceptor only queues when scaling from zero. So one warm pod takes all ten requests and handles them one at a time on a single event loop, and the overflow hits the proxy timeout at about 57s, which returns a 502.

Scaling was also too slow. The shared helm chart’s default topologySpreadConstraints: DoNotSchedule, on small nodes with about 1700m allocatable, meant each new pod waited for Karpenter to add a node. And a single burst is too short a signal to scale on. By the time the metric says “scale up”, the requests have already failed.

What to do instead

  • Provision capacity ahead of time: a warm pool, a KEDA cron scaler for business hours, or Lambda. This is why the production attribution path already runs on Lambda.
  • Get more out of each pod. We moved the CPU work into worker_threads, a pool of 4 per pod. Running the same burst against 3 pods instead of 10 gave 0/15 errors at the same ~12s median, and /ping stayed around 1.2s the whole time because the event loop was never blocked. Peak was about 2.4–2.8 cores per pod.
  • If the pool has to grow, scale on real busyness: KEDA with targetConcurrency: 4 to match the worker pool size, not 1.
  • The chart gained a topologySpread.whenUnsatisfiable value so services can opt into ScheduleAnyway.

Reactive autoscaling works for traffic that ramps up gradually. For a burst of slow calls, the capacity has to be running before the calls arrive.

← all writeups