IL-0088 UK Ministry of Justice · 2020-09-07kubernetesaws
On the Ministry of Justice Cloud Platform, teams get their ingress controllers from a shared ingress controller module. Version 0.1.0 of that module turned on the creation of a ValidatingWebhookConfiguration, so the ingress controllers' admission webhook began validating Ingress objects. Turning this on made the webhook a single point of failure for all ingress controller pods in the ingress-controllers namespace.
In the live-1 cluster one new 0.1.0 ingress controller could not be created because of AWS resource limits. From then on the webhook blocked new ingress rules: calls to the admission service of an ingress controller failed with "x509: certificate signed by unknown authority", and the Ingress API refused all POST requests. The team removed by hand all the extra admission controllers that 0.1.0 had created, took the admission webhook out of the module, released version 0.1.1 and upgraded every ingress module that was on 0.1.0. The report gives the time to repair as 3h 02m and the time to resolve as 3h 17m.
At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.
Creating an ingress failed in the validating admission webhook with this error: "Error from server (InternalError): error when creating "ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post offender-categorisation-prod-nx-controller-admission.ingress-controllers.svc:443/extensions/v1beta1/ingresses?timeout=30s: x509: certificate signed by unknown authority".
Symptom ? Deploys failing
Root cause ? Faulty deployment
The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.
–
Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.
same root cause · shared Ingress controller
same root cause · shared Ingress controller