IL-0073 Target · 2019-01-15 (published)kubernetesnetworking
The Target Application Platform (TAP) gives application teams one interface for running workloads across several hosting backends, including many Kubernetes clusters. Every workload deployed through TAP gets logging and metrics sidecars that ship data through Kafka, and a Consul agent sidecar that registers the workload as a service in Consul and joins the node to the Consul gossip mesh. One development cluster had been built much larger than the rest and hosted about 2,000 workloads.
When Kafka became intermittently available, all logging sidecars woke up at the same time. Each used little CPU, but together they put a high load on the Docker daemon of each node, and the nodes reported unhealthy. The scheduler moved workloads to healthy nodes, and because it does not spread load evenly, some receiving nodes could not take the extra CPU and failed too. During this rescheduling Kubernetes produced about 41,000 new nodes, which were quickly spun up and terminated. Their applications never started, but each pod lived long enough for its Consul agent to register and gossip. This raised CPU on Consul and slowed its responses, and the bigger mesh raised CPU for all workloads. Consul agents process only a limited number of messages from other nodes per loop, so the messages saying that nodes had left were not processed, and phantom nodes expired and returned in waves. The development Vault sealed itself and TAP deployments failed.
After days of investigation, the team enabled gossip encryption on Consul so that every message from the poisoned mesh was rejected immediately. That recovered Consul and Vault. The team then stabilized the deployment engine and redeployed the whole development environment to roll out encryption; instances not configured for it were terminated. Afterwards they found that Consul 1.2.3 included a merged change for large clusters and upgraded Consul in place, ending on 1.3.0 because of a UI bug in 1.2.3. A controlled scale-up and ungraceful scale-down of one workload showed nodes now leaving the gossip mesh and the service catalog quickly.
Production felt the same net effect, but its Kubernetes clusters were smaller and more sparsely packed, so the cascade did not happen there. Its gossip mesh was also poisoned, with much less load, and the Consul upgrade fixed it quickly. The TAP team treats the development environment as "prod0" and handled the outage at the highest level of severity.
An upgrade of the network subsystem under Target's datacenter OpenStack, meant to cause only a brief disruption, ran into problems and left network connectivity disrupted for several hours. Kafka, which runs on that OpenStack, became intermittently available.
In the one large development Kubernetes cluster, which held about 2,000 workloads, the Docker daemons came under high load and nodes reported to Kubernetes that they were unhealthy. Kubernetes moved workloads to the nodes that were still healthy, those nodes became unhealthy in turn, and the cycle kept going.
Consul slowed down. Requests to Consul had intolerable latency, and CPU use rose both on Consul and in every deployed workload. In the gossip mesh, nodes that were gone kept expiring and being added back in waves.
Then the systems that depend on Consul failed. The development Vault cluster, which serves secrets to applications, sealed itself because it could not reach Consul in time. The TAP deployment engine, which uses Consul to find its infrastructure backends, issue tokens to applications and configure load balancing, could not reach Consul either, and deployments began to fail.
Symptom ? Node NotReadyLoad imbalanceHigh latencyCredentials unavailable
Root cause ? Dependency failure
–
Smaller and more numerous clusters limited the damage: the smaller development clusters and production were not hit the same way as the one large cluster, and dev and prod on one cluster would have been far worse. A Docker daemon shared by many workloads is a brittle failure point. Per-workload logging and metrics sidecars were judged the right design; a shared cluster-level agent would almost certainly have made the problem worse and harder to fix.
–