Taking 41% off the AWS bill

PLACEHOLDER POST — seed data. Replace with your real writeup.

The bill had grown 70% in a year while traffic grew 20%. Finance asked for a plan. The plan was mostly to go and look.

Where the money went

1. EC2 for EKS nodes (48%): on-demand, sized for a peak that happened twice a year
2. NAT and cross-AZ data transfer (17%): chatty services spread across zones
3. Orphans (9%): unattached EBS volumes, old snapshots, idle load balancers

What we did

• Karpenter replaced cluster-autoscaler, with spot capacity for stateless workloads across 15 instance types
• Requests set from data: two weeks of p95 usage from Prometheus, applied through VPA in recommendation mode
• Topology-aware routing to keep traffic in its own zone
• Tag or delete: anything without an owner tag after 14 days was snapshotted and removed

# requested vs used CPU, per namespace — the chart that got buy-in
sum by (namespace) (kube_pod_container_resource_requests{resource="cpu"})
  / sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))

What didn’t work

Savings Plans before rightsizing. We almost committed to three years of capacity we were about to stop using.