IL-0089 UK Ministry of Justice · 2020-09-21kubernetesaws
The Cloud Platform team triggered a script meant to delete a test cluster. The kube context, the setting that decides which cluster command-line tools talk to, pointed at live-1, the production cluster, so the script deleted platform components there: the default ingress controller, prometheus-operator, logging, cert-manager, kiam and external-dns. Without the ingress controller some users could not reach their services, monitoring and Kibana were down, and certificates could not be issued.
The team restored the default ingress controller first, together with the two components it depends on. external-dns writes the address of the new NLB into Route 53 records, and kiam gives external-dns the AWS role it needs to do that. Once those three were back, services started to return. The report gives 19:05 as the repair time, 38 minutes after detection.
The Formbuilder services stayed down. The TXT records in Route 53 that external-dns uses to mark the records it owns had the wrong owner field, so external-dns did not update their A records to the new NLB. After the team corrected the owner field, external-dns pointed the Formbuilder records at the new NLB and the services came back. The team then restored the remaining components with a targeted apply and ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository. The incident was resolved at 21:41.
At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.
The platform's own tools went down too: Prometheus, Alertmanager, Grafana and Kibana could not be reached, and no new certificates could be created. The components that had disappeared from the cluster were the default ingress controller, prometheus-operator, logging, cert-manager, kiam and external-dns.
After the ingress controller was restored and most services came back, the Formbuilder services were still unreachable. Their Route 53 records still pointed at the old network load balancer (NLB), the one in use before the ingress controller was replaced.
Symptom ? Service outageResources deletedMonitoring outageStale endpoint routing
Root cause ? Operator error
The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.
–
–
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Ingress controller