Root cause ?
config-change8 cases · 2019 to 2025
IL-0022 Three overlapping routing and TPU bugs degraded model responses — Anthropic, 2025, sev3IL-0005 Telemetry rollout overloaded the control planes and broke service discovery — OpenAI, 2024, sev1IL-0009 A generated name collision made Argo CD delete the production environment — InfluxData, 2023, sev1IL-0019 A module upgrade shrank node disks and filled them with images — UK Ministry of Justice, 2023, sev3IL-0017 VPC CNI stopped allocating pod IPs on the live cluster — UK Ministry of Justice, 2023, sev3IL-0010 A dropped template delimiter made Argo CD delete every service worldwide — Skyscanner, 2021, sev1IL-0008 A validating webhook with a dead backend stalled a GKE master upgrade — Jetstack, 2019, sev1IL-0004 Pod priority defaults preempted production ingesters — Grafana Labs, 2019, sev2