IL-0091 UK Ministry of Justice · 2023-06-06kubernetesaws
The Cloud Platform team changed the instance type of the worker nodes with Terraform. The change recycled all nodes at the same time, so every pod on them was deleted at once and user services went down for a few minutes. At 10:30 the team found that the nodes were all being recycled together during the instance type change.
Users reported most services back up by 10:30, but the DPS service stayed unavailable until 11:21 because its users could not authenticate. The incident was declared at 11:00 and repaired at 11:21; the report gives 55 minutes as both the time to repair and the time to resolve. The team opened a ticket with AWS asking how to change the instance type without an outage and planned to write the procedure down as a runbook.
At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.
By 10:30 users reported that their services were running again. At 10:50 a user reported that the DPS service was down because people could not authenticate into it; the DPS service was reported back up at 11:21.
Symptom ? Service outage
Root cause ? Configuration change
Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.
–
–
same root cause · shared Terraform, Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Terraform
same root cause · shared Terraform
same root cause · shared Node pool