It looked like a rate limit
cert-manager kept failing to issue one certificate, with ACME 404s. It looked like Let's Encrypt rate limiting. Restarting the cert-manager controller fixed it, and our deleting and recreating the Certificate had only made it worse.
- Date
- 2026-06-03
- Type
- note
- Impact
- The beta environment's SSO hostname couldn't get a TLS certificate, which blocked beta auth from being onboarded.
A new Certificate for the beta SSO hostname kept failing: 22 attempts. The ACME errors
were strange: No such authorization, No such challenge, No order for ID <n>.
cert-manager created an order, and then the order couldn’t be fetched.
What actually happens
Setup: a ClusterIssuer against Let’s Encrypt, DNS-01 through a GoDaddy webhook. DNS-01
validates with a _acme-challenge TXT record, so ingress and routing don’t matter.
The first theory was a Let’s Encrypt limit. LE allows 5 failed validations per account, per hostname, per hour. We had deleted and recreated the Certificate about 5 times, and every delete abandons an ACME order partway through. That produces failed validations and resets the sliding window, so the theory fit.
A test disproved it as the whole story. A different hostname, on the same account, issuer and webhook, issued in about 3 minutes. The webhook logs showed TXT records being created. The account, DNS and webhook were all healthy. The problem was specific to this one hostname.
Then:
kubectl rollout restart deploy cert-manager -n cert-managerThe certificate issued right away.
Why
The best explanation is stuck state inside the cert-manager controller: its in-memory ACME account, nonce and order cache kept pointing at orders the ACME server no longer had. Our deletes kept feeding it more dead orders.
I can’t prove this beyond “the restart fixed it”. But a real rate limit would have kept going after a restart, and this didn’t. Both problems produce the same ACME 404s, and the way you’d try to fix the first one makes the second one worse.
What to do instead
- On ACME
malformed404s, restart the cert-manager controller first. Then leave the Certificate alone and let it retry once. - At most one clean recreate. After that, stop. More deletes create more failed validations and can trip the LE limit for real.
- To tell a hostname problem from an account problem, issue a known-good hostname on the same issuer. It takes about 3 minutes.
- Before blaming DNS, check the webhook logs for the TXT create/update line.
Deleting and recreating an ACME Certificate is not a retry. It’s a new, abandoned order, and Let’s Encrypt counts it against the hostname.
← all writeups