The 17th rule never shipped
Gateway API caps an HTTPRoute at 16 rules. Our 19-rule route failed ArgoCD sync for two days while the app still showed Healthy.
- Date
- 2026-09-02
- Type
- incident
- Duration
- ~2 days
- Impact
- A newly onboarded service in the test environment returned a gateway 404 for every request, although its pod, Service and Deployment were all fine.
A new service was deployed but unreachable. Every request through the gateway got a
plain page not found. Curl straight to the pod IP returned 405 Method Not Allowed,
which is the right answer to a GET on a POST-only endpoint. So the app was fine, and
the route was in git. ArgoCD showed the routing app as Healthy.
Timeline
| Time (UTC) | Event |
|---|---|
| Aug 31 | New /campaignmapping/ rule appended to an existing HTTPRoute and pushed |
| Aug 31 13:39 | ArgoCD sync fails, retries 5 times, gives up. App stays Healthy / OutOfSync |
| Sep 2 | Service reported unreachable. Pod-direct curl works, gateway returns 404 |
| Sep 2 ~12:57 | kubectl get httproute has no such rule. Sync error found in app conditions |
| Sep 2 12:58 | Route split into a new HTTPRoute object, committed |
| Sep 2 12:59 | ArgoCD syncs about 51s after the push. New route object created |
| Sep 2 13:00 | Gateway now returns the app’s own auth error. Traffic is reaching the pod |
The actual cause
Routes for the test environment live in one file with several numbered HTTPRoute
objects: microservices-test-1 through -5. The numbering exists only because of a
limit nobody had written down. The Gateway API CRD validates spec.rules to at most
16 items.
Adding the new rule took microservices-test-5 to 19. The API server rejected the
whole object:
spec.rules: Too many: 19: must have at most 16 itemsThis is not a Traefik limit. It’s CRD schema validation, so any Gateway API implementation behaves the same way.
ArgoCD’s reporting made it easy to miss. Health covers resources that exist in the
cluster, and those were all fine, because the old version of the route was still
applied. The failure only showed up as OutOfSync plus a SyncError in
.status.conditions. Nothing alerted on it. For two days git said one thing and the
cluster did another. Anything else added to that object in the meantime would have
failed the same way.
What changed
- Moved the last three rules into a new
microservices-test-6HTTPRoute with the sameparentRefsand hostnames.-5now has exactly 16 rules. - Onboarding a service now includes counting the rules in the target HTTPRoute. At 16, start a new numbered object.
- When traffic doesn’t reach a pod, check ArgoCD sync status and conditions before
debugging the app, the Service or the Deployment.
Healthyalone doesn’t show sync failures.
What didn’t work
We started at the app. Checked the Flask routes, PATH_ALIAS and the probe paths, and
went looking for a /ping endpoint that turned out not to exist. The 405 from the
pod IP had already ruled out the app. We should have gone straight to the layer between
the gateway and the pod.
An app can show Healthy in ArgoCD while its sync keeps failing. If you alert on one, alert on the other too.
← all writeups