IL-0060 Reddit · 2023-03-14kubernetesawsnetworking
Reddit's cluster for the legacy stack was one of several bespoke clusters built with kubeadm. Years earlier, a predecessor of the Compute team had configured Calico, the cluster's network plugin, to use route reflectors: instead of every node peering with every other node in a full mesh, the control plane nodes act as reflectors and all other nodes peer only with them. The route reflector nodeSelector and peerSelector matched the label node-role.kubernetes.io/master. Kubernetes renamed the role to control-plane in the 1.20 series and removed the master label in 1.24. When the first control plane node was upgraded, calico-node across the cluster began dropping routes to it, as expected, but it also began dropping the routes for every other node; that happened two seconds before the site broke. Cluster networking failed, and with it most user flows. The route reflector setup could only be edited through Calico's own CLI, was committed nowhere, and everyone who knew about it had left; it was only found during the postmortem.
Kubernetes has no supported downgrade path, so rolling back meant restoring the control plane from the backup taken at the start of the upgrade. The restore procedure was years old, written for an end-of-life Kubernetes version and for Docker rather than CRI-O, and had never been run against a production cluster. After more than two hours of trying to fix forward, the team made the call to restore. All worker nodes were terminated first, which took about 20 minutes of API calls. The procedure was rewritten live as commands failed, and the restore to the first control plane node succeeded. The cluster autoscaler came back and was switched off to regain control. Launching the two replacement control plane nodes was delayed because AWS had no capacity for that instance type. The new nodes then could not join, because the backup had been taken on a different control plane node from the one restored, so the TLS certificates did not match the hostname; the team generated new certificates.
Traffic was brought back gradually, at 1%, 5%, 10%, 20%, 35%, 55%, 80% and 100%, to avoid a thundering herd against cold caches and scaled-down services, and some legacy services that had been stopped by hand were started again. The outage lasted 314 minutes. Reddit's modern service API layers stayed up, but the legacy cluster sits on the most critical node of the dependency graph.
About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.
The affected cluster had lost all its metrics, because Reddit's metrics are Kubernetes native, so responders had little to look at. DNS was failing in a strange way: records in Consul and in-cluster DNS names would not resolve, while public DNS records resolved normally.
Over the next two hours more signs appeared. Pods took a very long time to start and stop, container images under 100MB took minutes to pull over a multi-gigabit link, and control plane logs were heavy without an obvious error. Calico pods were unhealthy: calico-kube-controllers was stuck in ContainerCreating, and deleted calico-typha pods were not recreated. API server logs showed many timeouts on write operations, which turned out to be timeouts calling the Open Policy Agent admission webhook. API server log volume had jumped fivefold at 19:04:49 UTC.
The responders deleted the stuck pod, restarted calico-typha, did a rolling restart of the control plane, then switched the entire control plane off and on, and finally deleted the OPA webhook configurations. The webhook timeouts stopped at once, but the cluster did not recover.
Symptom ? Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag
Root cause ? Breaking change in an upgrade
The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.
Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.
same root cause · shared Admission webhook
same root cause · shared Admission webhook