IL-0098 2i2c · 2026-03-31kubernetesnetworking
2i2c had recently migrated its clusters from the ingress-nginx controller to nginx-ingress, and the two treat several Ingress objects that share the same host differently. To satisfy the ACME issuer flow, cert-manager creates a temporary Ingress for the host it needs a certificate for. Under nginx-ingress the hub's existing Ingress for that host took priority, the challenge request was answered with a 301, and certificate orders stayed pending. The problem showed up when a deployment needed a new certificate, on at least the 2i2c-aws-us and awi-ciroh clusters.
An incident was declared at 16:00 on 31 March 2026. The night before, the engineer had tried cert-manager's annotations for editable Ingresses without clear success. At 16:43 the engineer tried the master-minion Ingress model instead; it issued the certificates, but it requires the paths to be set in the minion Ingress, which would mean splitting each Ingress in two, and that was hard to do in a general way. The engineer went back to editable Ingresses: annotating the Ingress objects that need TLS so that cert-manager may change them in place (edit-in-place) restored certificate issuance. The incident was resolved at 17:16, 1 hour 15 minutes 39 seconds after it was declared.
An attempt to launch a new hub failed because its ingress could not obtain a new TLS certificate. The day before, the same engineer had recreated the support deployment on the 2i2c-aws-us cluster and seen its TLS certificate orders stuck pending.
The nginx-ingress logs showed 301 responses, which the engineer suspected were HTTP-to-HTTPS redirects. The first question was whether redirecting to SSL had become a new default, but the documentation for both nginx-ingress and ingress-nginx suggested it had not. Looking at how the ACME issuer flow works, the engineer then found evidence that the hub's own Ingress was taking priority over the temporary Ingress that cert-manager creates for the challenge, which likely produced the 301.
The same failure then appeared on the awi-ciroh cluster, where a decommissioned workshop hub had just been redeployed and needed new certificates. The JS2 cluster, recently redeployed with the new ingress controller, had shown no sign of it.
Symptom ? Misrouted requestsDeploys failing
Root cause ? Breaking change in an upgrade
An attempt to launch a new hub failed; the engineer had seen certificate orders stuck pending on another cluster the day before and declared an incident.
Swapping one ingress controller for another can change how Ingress objects that share a host are combined. cert-manager's temporary challenge Ingress depends on that behaviour, so the breakage shows up when a deployment needs a new certificate.
2i2c notes that it got lucky: it had already learned of this possible cause while testing on an internal cluster, and changing the cert-manager configuration was quick.
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller