IL-0010 Skyscanner · 2021-08-25kubernetesaws
The edit lost the template delimiters, so templating no longer ran and every namespace definition built from that file came out corrupted. Argo CD reconciled toward a configuration with no valid namespaces and deleted all 478 services across every availability zone and region. Recovery meant restoring the correct configuration per cluster and letting reconciliation rebuild each one, with a single region and the critical services first; traffic was served again after about four and a half hours and the rest came back over two days. A backup was harder to reach than expected because of an IAM policy change.
The website and the mobile apps stopped working for everyone at once. The GitOps tool was removing services from every cluster in every region, and the product stayed down for about four and a half hours.
Root cause ? Configuration change
Trigger A small, peer-reviewed edit to a root-level templating file of the provisioning system, intended as a no-op, that was pushed to every region at once.
unknown
Configuration with templating and logic is code and needs tests. Failure scenarios and runbooks were not pessimistic enough, a backup is only proven by a restore, and an immediate global rollout of a rarely touched file is a hazard.
IL-0009 A generated name collision made Argo CD delete the production environmentsame root cause · shared Argo CD