The night DNS lied to us

PLACEHOLDER POST — replace with your real writeup. The structure below is the shape that works: what broke, the timeline, the actual cause, what changed.

At 03:14 the pager went off for checkout-api 5xx rate. Every dashboard we looked at was green. That contradiction cost us the first ninety minutes.

What people saw

Roughly a third of checkout attempts failed with a generic error. The other two thirds were completely fine, which is what made it so hard to believe.

Timeline

Time	Event
03:14	Paged on checkout-api 5xx rate
03:22	Dashboards green, assumed false positive
04:05	Confirmed failures reproducible from one AZ only
05:30	Found NXDOMAIN cached for an internal service name
06:40	Flushed cache, error rate returned to baseline
07:26	Incident closed

The actual cause

A brief control-plane blip caused the internal DNS resolver to receive NXDOMAIN for a service that genuinely existed. The negative cache TTL was 300s, but the client library cached it for the lifetime of the process.

# the tell — same query, different answer per node
for n in $(kubectl get nodes -o name); do
  kubectl debug "$n" -it --image=busybox -- nslookup payments.internal
done

The pods that were restarted after the blip were fine. The pods that had been running for three weeks held onto a lie indefinitely.

What changed

• Negative cache TTL pinned to 30s at the resolver
• Readiness probe now performs an actual dependency resolution
• Alert on divergence between nodes for the same query, not just error rate

The lesson I actually took away: when dashboards and users disagree, the users are right and the dashboard is measuring the wrong thing.