← all writeups
CI runners that scale to zero
GitHub Actions runners on Kubernetes with ARC: no idle VMs overnight and no queue in the morning.
- Date
- 2026-02-09
- Type
- build
- Duration
- 2 weeks
- Impact
- Runner cost down 63%. p95 queue time from 7 minutes to 20 seconds.
PLACEHOLDER POST — seed data. Replace with your real writeup.
We had 30 static runner VMs. At night, 30 were idle. At 10am, 30 weren’t enough and jobs queued for seven minutes.
What we built
- Actions Runner Controller scale sets, one per workload class (
small,docker,gpu) - Ephemeral runners: each job gets a fresh pod, so no state leaks between jobs
- A dedicated spot node pool managed by Karpenter, which scales to zero when there are no jobs
spec:
minRunners: 0
maxRunners: 80
template:
spec:
containers:
- name: runner
resources: { requests: { cpu: "2", memory: 4Gi } }The hard part: Docker builds
Docker-in-Docker needs privileged pods. We moved image builds to BuildKit running rootless as a sidecar, with a shared registry cache. Cold builds got slower and warm builds got much faster.
Numbers after a month
| Metric | Before | After |
|---|---|---|
| p95 queue time | 7m | 20s |
| Monthly runner cost | 100% | 37% |
| Flaky “dirty runner” failures | ~15/week | 0 |
← all writeups