← all writeups
Out of ports on the NAT gateway
A retry storm against a third-party API opened 55,000 connections through one NAT gateway. Everything else behind it lost the internet.
- Date
- 2025-06-08
- Type
- incident
- Severity
- SEV2
- Duration
- 1h 12m
- Impact
- Outbound calls from private subnets in one AZ failed intermittently; payments and email delayed.
PLACEHOLDER POST — seed data. Replace with your real writeup.
The symptom looked random: payment webhooks timing out, emails queueing, image fetches failing. It only happened in one availability zone. Nothing linked those services except the route out to the internet.
Timeline
| Time | Event |
|---|---|
| 09:40 | Shipping-rates provider starts responding slowly |
| 09:44 | Our client retries with no backoff and no connection reuse |
| 09:51 | ErrorPortAllocation climbing on nat-az1 |
| 10:20 | Traced 90% of connections to the shipping service |
| 10:31 | Circuit breaker forced open, connections drain |
| 10:52 | Error rate normal, incident closed |
The actual cause
A NAT gateway allows about 55,000 simultaneous connections to a single destination IP and port. The shipping client created a new HTTP client per request, so every retry opened a fresh connection. Each retry then waited out the full 350-second idle timeout.
What changed
- Shared HTTP client with keep-alive and a bounded connection pool
- Exponential backoff with jitter, plus a circuit breaker per upstream
- CloudWatch alarm on
ErrorPortAllocation > 0 - One NAT gateway per AZ, with a dashboard showing which service owns the connections
Retries without backoff don’t make you resilient. They just make a small outage bigger.
← all writeups