Incident Library?

Incident Library?

Contributing factor ?

Unknown limit 한계 미인지 · `limit-unknown`

4 cases · 2019 to 2020

Root cause ?Capacity exhaustion 3 Software defect 1

Component ?Node pool 2 Cluster autoscaler 1 conntrack table 1 CoreDNS 1 Container CPU limit 1 Client-side DNS resolver and cache 1 HAProxy 1 kube-proxy 1 kubelet 1 Linux kernel 1 Container memory limit 1 VPC subnet or pod address range 1

Symptom ?High error rate 3 Scale-up failure 1 Connection resets 1 CPU throttling 1 DNS resolution failures 1 High latency 1 Intermittent connection failures 1 Intermittent timeouts 1 Monitoring outage 1 IP address exhaustion 1 OOMKilled 1 Pods Pending 1 Probe failures 1

Cases

IL-0012 CPU limits and a kernel scheduler bug stalled multi-threaded services — Omio, 2020, sev3
Root cause Software defect
IL-0013 Connection resets from a full conntrack table on GKE nodes — loveholidays, 2020
Root cause Capacity exhaustion
IL-0015 A GKE pod address range ran out at 256 nodes — loveholidays, 2020, sev3
Root cause Capacity exhaustion
IL-0001 Cluster-wide DNS outage after CoreDNS ran out of memory — Zalando, 2019
Root cause Capacity exhaustion

View on the graph