IL-0072 Nordstrom · 2017-12-15 (published)kubernetesaws
Nordstrom ran Kubernetes on AWS with a five-node control plane: five etcd instances, five API servers, and schedulers and controller managers in high-availability mode. In etcd, writes are always consistent, but a read is answered from the local member's data unless the client asks for a quorum read. When members fall far behind each other, those local answers are stale, and long replication lag can also cause repeated leader elections. The Kubernetes API server in its default configuration did not request quorum reads, and the high-availability documentation did not mention the setting until October of the year of the talk.
In this incident etcd was hit by high write latency. The API servers returned stale data, so kubectl output, the controller manager and service endpoints flipped between two states. The speakers stress that this was stale data, not a breakdown of etcd's consensus.
The outage on the primary production cluster lasted about four hours, until service was restored on a replacement cluster. Confusion and reluctance to replace the cluster added to the time, because teams used features that are hard to move. Each persistent volume can be attached to only one node, so volumes had to be freed from the old cluster one by one before they could bind in the new one. Some teams had pointed DNS records at the generated names of their Kubernetes load balancers and had to update them by hand to the new load balancers.
The fix is to make the API server request quorum reads. etcd 3.1 makes quorum reads much cheaper because they no longer hit disk, and quorum reads became the default in Kubernetes 1.9. The incident happened during working hours with the full team present, and the root cause analysis led to changes in the team's code and procedures.
A user reported that pods were starting up and then vanishing. When the cluster operators looked, kubectl get pods and kubectl get nodes returned two different sets of data that alternated back and forth, as if there were two opinions of the cluster's state.
The control loops acted on the conflicting data. The controller manager, querying API servers and getting alternating answers, kept trying to converge and spun up and terminated thousands of pods. Service endpoints thrashed, and the ingress controller, responding to the conflicting signals, did very bad things to the traffic that should have flowed through it.
It was a full outage of the primary production cluster: not just down, but violently wrong. The team spent a long time troubleshooting before deciding to replace the cluster.
Symptom ? Stale or inconsistent API readsStale endpoint routingService outage
Root cause ? Software defect
A user reported pods starting up and vanishing.
etcd is extremely sensitive to write latency; once the database sync time crosses double digits of milliseconds, stale reads or badly degraded performance follow. Without quorum reads, a consistent store can still hand out stale data.
Documentation for a fast-moving system changes, so read it again regularly. Moving workloads to a new cluster is hard when they hold persistent volumes or when DNS records point at generated load balancer names.
same root cause · shared Kubernetes API server, etcd
same root cause · shared Kubernetes API server, etcd
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Kubernetes API server
same root cause · shared Ingress controller
same root cause · shared etcd
same root cause · shared Persistent volumes and stateful sets
same root cause · shared Ingress controller
same root cause · shared Ingress controller
same root cause · shared Kubernetes API server