← all writeups
Every certificate expired at midnight
cert-manager had been failing renewals silently for sixty days. We found out when the ingress started serving expired certs to every client.
- Date
- 2026-08-02
- Type
- incident
- Severity
- SEV1
- Duration
- 2h 05m
- Impact
- All public HTTPS endpoints returned certificate errors for ~95 minutes.
PLACEHOLDER POST — seed data. Replace with your real writeup.
At 00:00 UTC the wildcard certificate for every public hostname expired. Browsers refused to connect, mobile clients hard-failed, and the status page (behind the same ingress) went down with everything else.
Timeline
| Time | Event |
|---|---|
| 00:00 | Certificate *.example.com expires |
| 00:04 | Synthetic checks fail across all regions |
| 00:19 | Confirmed expiry, not a CA outage |
| 01:10 | Found the Challenge objects stuck in pending since June |
| 01:42 | Manual DNS-01 issuance succeeds, secret rotated |
| 02:05 | All endpoints healthy, incident closed |
The actual cause
A DNS provider API token was rotated during a security cleanup. cert-manager’s
DNS-01 solver started getting 403 on every renewal attempt, logged it at info,
and retried forever. The old certificate kept working for sixty days, so nothing
looked wrong.
kubectl get certificates -A -o wide | grep -v True
kubectl describe challenge -n ingress # "403 Forbidden" every 15 minutes since JuneWhat changed
- Alert on
certmanager_certificate_expiration_timestamp_secondsbelow 21 days - Alert on any
CertificatewithReady=Falsefor more than an hour - Solver credentials moved into the secret-rotation runbook, with a test issuance step
Expiry was never the problem. Sixty days of silence was.
← all writeups