One t3.nano between us and the internet

Nothing broke. We were testing things and mapping how the network actually worked. The VPC Terraform showed that all outbound traffic went through one t3.nano, with iptables MASQUERADE set up by hand and ignore_changes = [ami]. Every private subnet in every AZ depended on it.

The constraint

• One NAT instance in one public subnet, in one AZ.
• One shared private route table for all four private subnets (AZ a–d). Its 0.0.0.0/0 route points at that instance’s ENI.
• No auto-recovery and no failover. If the instance or its AZ goes down, all four subnets lose egress.
• It’s a t3.nano, so throughput depends on burst credits.
• Three of the four AZs already pay $0.01/GB cross-AZ just to reach it.

The other constraint is money. A NAT Gateway in every AZ is the standard answer, and the expensive one.

The options

1. Managed NAT Gateway per AZ. No maintenance at all, but about $32/month per AZ plus $0.045/GB processed.
2. fck-nat in a one-instance ASG per AZ. t4g.nano Graviton. On boot each instance claims its EIP and rewrites its AZ’s route table to point at its own ENI, so it recovers on its own in about 1–2 minutes. About $6–8/month for two.
3. Two static NAT instances, reusing the existing AL2023 + iptables config. The smallest change to apply. No self-healing.

Any of these also needs the shared route table split into one per private subnet, and the S3 gateway endpoint attached to all of them. Otherwise those subnets lose their free S3 path.

What we chose and why

Two NAT AZs, a and b. Subnets c and d route to one of them cross-AZ (nat_index = az_index % nat_az_count).

For a first version, option 3: two static instances, the lowest-risk change. fck-nat’s self-healing is the next step, to be proven in beta first.

Adding a second NAT alone doesn’t solve it, though. A node’s outbound traffic follows its subnet’s route table, not the pod, and Karpenter has no idea whether a NAT is healthy. If AZ-a dies, pods can reschedule onto an AZ-c node, and AZ-c routes to the dead NAT in AZ-a. So the plan also pins the microservices NodePool to the two NAT AZs:

- key: "topology.kubernetes.io/zone"
  operator: In
  values: ["us-west-2a", "us-west-2b"]

Now, if AZ-a goes down, its NAT and its compute go down together, pods move to AZ-b, and AZ-b’s NAT is still up. The stateful services already live in a and b, so c and d weren’t holding anything important.

Open questions

• Never applied. This was a design exercise. Both changes (Terraform, and the NodePool change in the GitOps repo) exist only as commits on a local branch. terraform validate passes. terraform plan was never run, so everything below is what I’d check first.
• Route table churn. Splitting the route table destroys the shared one, so there will be a short egress outage during the apply. moved {} blocks can keep the old table as [0] and halve the churn. It still needs a maintenance window.
• Unintended replacements. On this VPC, module-level depends_on has triggered EKS replacement cascades before. The plan output needs a line-by-line read.
• Self-healing. With static instances, an instance AWS terminates stays gone until someone runs terraform apply. The other AZ keeps serving, but that’s not recovery.
• System pool. Only the microservices NodePool is pinned. The system NodePool can still land in c or d.

Nobody treats a NAT as infrastructure until it’s the only thing between a whole VPC and the internet.