IL-0005 OpenAI · 2024-12-11kubernetes
The telemetry service's configuration made every node issue Kubernetes API calls whose cost grew with cluster size. In the largest clusters that overwhelmed the API servers. Service discovery depends on DNS backed by the control plane, so once node-level DNS caches expired, services lost each other. Staging had not shown the problem because the effect only appears at scale, and the cache delay let the rollout finish before failures were visible. Recovery was slow because the overloaded control planes also blocked the operators: they shrank clusters, blocked admin API access, scaled API servers up and removed the service. Everything was back after a little over four hours.
Alerts fired and within minutes every product was degraded or down. The API servers of the largest clusters stopped answering, and as cached DNS records expired, services could no longer find each other.
Symptom ? Service outageControl plane unresponsiveDNS resolution failures
Root cause ? Configuration change
Trigger A new telemetry service was rolled out to every cluster within half an hour.
Existing monitoring alerted a few minutes before customers saw impact.
Tests missed an effect that only appears with cluster size, DNS caching hid the failure until the rollout was complete, and the data plane was less independent of the control plane than assumed.
IL-0008 A validating webhook with a dead backend stalled a GKE master upgradesame root cause · shared Kubernetes API server