IL-0090 UK Ministry of Justice · 2021-06-09kubernetes

After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change

The ModSec ingress controller registers a validating admission webhook, which the Kubernetes API server calls before it accepts any Ingress create or change. After the module was upgraded to v3.33.0, the API server called the webhook at the service name modsec01-nx-modsec-admission, but the TLS certificate the webhook served was valid only for modsec01-nx-controller-admission. The API server therefore could not call the webhook, and every Ingress create or change was rejected, including those in all user application deployments.

The team rolled the ModSec ingress controller module back to v0.0.7. The incident was repaired at 13:46, 31 minutes after the first report, and resolved at 13:58. The review asks why the test cluster did not show the problem, and notes that neither the old nor the new ModSec module version worked on the cluster named live.

Severity: –
Scope: all Ingress create and update calls on the cluster, and with them all user application deployments
Users affected: all users deploying applications on the Cloud Platform
Region: –
Time to detect: –
Time to resolve: 43m

Symptoms

At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

The error was: kind Ingress: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://modsec01-nx-modsec-admission.ingress-controllers.svc:443/networking/v1beta1/ingresses?timeout=10s: x509: certificate is valid for modsec01-nx-controller-admission, modsec01-nx-controller-admission.ingress-controllers.svc, not modsec01-nx-modsec-admission.ingress-controllers.svc

Every Ingress API call was blocked. No new Ingress could be created and no change to an existing Ingress could be deployed, which stopped all user application deployments. The failures started right after an upgrade of the ModSec ingress controller module that had appeared to deploy successfully.

Symptom ? Deploys failing

Root cause

Root cause ? Breaking change in an upgrade

Component ?Admission webhook Ingress controller

Contributing factor ?Unrepresentative staging

Trigger: The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

Detection

A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

Remediation

rolled back the ModSec ingress controller module to v0.0.7

Action items

Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972) unknown
Add a smoke test that checks the alerts in Alertmanager (ticket #2973) unknown
Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974) unknown
Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975) unknown

Lessons

–

Sources

https://runbooks.cloud-platform.service.justice.gov.uk/incidents/2021-06-09-unable-to-create-new-ingress-rules.html · runbook

Related cases

IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates2i2c · 2026
same root cause · shared Ingress controller
IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applicationsUK Ministry of Justice · 2026
same root cause · shared Ingress controller
IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024
same root cause · shared Ingress controller
IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replacedHoneycomb · 2024
same root cause · shared Ingress controller
IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in productionSunwoo Park · 2023
same root cause · shared Admission webhook
IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023
same root cause · shared Admission webhook