Incident Library?

Incident Library?

Contributing factor ?

Missed in review 리뷰 누락 · `review-missed`

4 cases · 2019 to 2023

Root cause ?Configuration change 4

Component ?Argo CD 2 Configuration templating 1 CoreDNS 1 Node launch template 1 Node root disk 1 Node pool 1 Persistent volumes and stateful sets 1 Pod priority and preemption 1 AWS VPC CNI 1

Symptom ?Service outage 3 High error rate 2 Pod eviction 2 Resources deleted 2 Image pull failures 1 IP address exhaustion 1 Node disk full 1 OOMKilled 1

Cases

IL-0009 A generated name collision made Argo CD delete the production environment — InfluxData, 2023, sev1
Root cause Configuration change
IL-0019 A module upgrade shrank node disks and filled them with images — UK Ministry of Justice, 2023, sev3
Root cause Configuration change
IL-0010 A dropped template delimiter made Argo CD delete every service worldwide — Skyscanner, 2021, sev1
Root cause Configuration change
IL-0004 Pod priority defaults preempted production ingesters — Grafana Labs, 2019, sev2
Root cause Configuration change

View on the graph