aman
← all writeups

One NAT instance per AZ, not one per VPC

Every private subnet in the cluster's VPC left through one t3.nano in one AZ. I split it into two NAT instances, each with its own route table, and skipped the managed NAT Gateway on purpose.

Date
2026-07-03
Type
build
Impact
Before, losing one instance or one AZ cut outbound access for every pod: image pulls, external APIs, all of it. Now it cuts off half the private subnets. No outage forced this. It came out of a review before client onboarding.

The EKS cluster’s NAT was a single t3.nano running AL2023 and iptables, in one public subnet. One private route table sent every private subnet’s 0.0.0.0/0 to it. There was no health check, no replacement, and no second route. Nothing had broken yet. That was the only thing it had going for it.

Where the traffic went

I checked capacity first, expecting to find a saturated instance. There wasn’t one. Over 30 days CloudWatch showed about 127 GB in and 127 GB out, a peak around 21.4 Mbps, and CPU credits that never ran out.

In and out being almost equal was the clue. It was mostly ECR image pulls. ECR layer blobs live in AWS-managed S3 and reach nodes through presigned URLs. From a private subnet, every layer goes through the NAT. An S3 gateway endpoint fixes that at the route-table level. It costs nothing and needs no IAM on the bucket, because the signature is in the URL.

So the NAT was not too small. It was a single point of failure.

What we built

The first plan was a managed NAT Gateway per AZ, at about $95/month each. That buys HA and headroom we clearly didn’t need at 21 Mbps. I went cost-first instead:

  • nat_az_count (default 2) splits the one shared private route table into one table per AZ.
  • Private subnets are round-robined across the tables: a and c to table 0, b and d to table 1.
  • There is one NAT instance per route table, count = length(var.private_route_table_ids), in us-west-2a and us-west-2b. Each has its own EIP and a default route to its own ENI.
  • The same AL2023 + iptables MASQUERADE config I already trusted, with source_dest_check = false.
  • The S3 gateway endpoint is attached to every private route table, not just the old one.

How it works

Each route table has exactly one NAT. If 2a’s instance dies, subnets a and c lose egress. Subnets b and d keep going through 2b. The blast radius is half the VPC, not all of it.

What didn’t work

  • No self-healing. These are static instances with no ASG. If AWS retires one, someone has to run terraform apply. The other AZ keeps serving until then. The planned next step is fck-nat in an ASG.
  • Round-robin isn’t AZ-local. Subnet c routes through the NAT in 2a, so that egress crosses AZs, and an outage in 2a takes out c too. Two NATs for four subnets was a cost call, not a topology one.
  • Terraform fought the change. An earlier edit to the VPC module made the plan want to replace the NAT instance for no real reason. That was a module-level depends_on. It has its own post.

A NAT instance isn’t the problem. One NAT instance for the whole VPC is.

← all writeups