aman
← all writeups

110 pods, zero reservations

We raised Karpenter maxPods to 110 for prefix delegation and set no kubelet reservations. A packed 4 GB spot node ran out of memory at the OS level and took the platform stack down with it.

Date
2026-05-28
Type
incident
Impact
The monitoring, ingress and auth pods on one system node became unreachable when the node wedged. Kubernetes couldn't evict or reschedule them because the kubelet had stopped responding.

A system node stopped heartbeating. EC2 said running. SSM couldn’t reach it. The kubelet was gone, so nothing was evicted, nothing was rescheduled, and everything on the node was stuck. That node ran monitoring, ingress and auth.

Timeline

Date Event
May 22 VPC CNI prefix delegation enabled. Managed node group t3.medium goes from 17 to 110 max pods
May 22 Karpenter t3a.medium nodes stay at 17. kubelet.maxPods: 110 set explicitly on EC2NodeClass
May 28 Spot c6a.large system node wedges at 95% memory requests / 166% limits
May 28 Root cause: maxPods 110 with no kubeReserved, systemReserved or evictionHard

The actual cause

This came in two steps, and the first fix set up the second failure.

Step one. We enabled ENABLE_PREFIX_DELEGATION on the VPC CNI. Managed node groups (AL2023 nodeadm) read that setting and recomputed max pods. Karpenter doesn’t. It computes --max-pods from the old per-ENI IP formula and never looks at the aws-node DaemonSet. So we set maxPods: 110 on every EC2NodeClass.

Step two. That EC2NodeClass had no kubelet reservations. The kubelet reserves memory for itself and the OS, and the EKS formula scales with pod count: 255Mi + 11Mi × maxPods. At 110 pods that’s about 1465Mi, over a third of a 4 GB node. We reserved nothing.

The scheduler did what it was told. It packed the c6a.large to 95% of memory requests, with limits at 166%. When real usage passed physical memory, there was no headroom for eviction to act. The OS itself ran out of memory, the kubelet died, and the node wedged. Losing a single pod would have been fine. We lost the whole node.

# EC2NodeClass — maxPods and its memory reservation belong together
kubelet:
  maxPods: 110
  kubeReserved:
    memory: 1465Mi   # 255Mi + 11Mi * 110

What changed

  • Rule: any EC2NodeClass with a high maxPods also sets kubeReserved, systemReserved and evictionHard, sized to that maxPods.
  • In Karpenter v1, maxPods lives on the EC2NodeClass. A different pod density per pool means a separate EC2NodeClass per pool.
  • The microservices pool keeps high density but no longer allows medium (2 GB) instances. Reserving about 1.46Gi out of 2 GB leaves nothing for pods.
  • Planned, not done yet: the system pool moves to low maxPods (~40), m family and on-demand. Ten heavy infra pods on spot were never a good fit for maxPods 110.

What didn’t work

Treating maxPods as a networking setting. It controls memory just as much.

Raising pod density without raising reservations doesn’t give you more capacity. It lets the scheduler overcommit the node until the OS runs out of memory.

← all writeups