IL-0008 Jetstack · 2019-09-05 (published)kubernetesgcp
An Open Policy Agent validating webhook had been installed cluster-wide, on every namespace and resource, and its backend was not answering. When the second master came up during the upgrade, the API server's start-up hook could not write a ConfigMap through that webhook, failed its health check and crash-looped, which halted the upgrade. With the API server flapping, kubelets could not report health, and node auto-repair kept recreating nodes. Deleting the webhook during a window of API access restored the cluster. The team then limited the webhook to the namespaces and resources it governs and gave the agent a liveness probe.
A Terraform-driven control-plane upgrade did not finish in its twenty-minute window and a rerun reported the API server unhealthy. The API server timed out on and off so nobody could deploy, and then every node began to be destroyed and recreated in a loop, taking every tenant down.
Symptom ? Upgrade stallControl plane unresponsiveNode recreation loopService outage
Root cause ? Configuration change
Trigger A planned control-plane upgrade of a regional development cluster shared by many teams.
The Terraform pipeline timed out, and a rerun returned an explicit API-server-unhealthy error; the sequence was reconstructed with the provider's support.
Alerting on API server latency would have shown writes slowing as soon as the webhook went in. Every workload, including the admission controller, needs probes; the tutorial install skipped the chart that already had one.
IL-0005 Telemetry rollout overloaded the control planes and broke service discoverysame root cause · shared Kubernetes API server