IL-0191 Grafana Labs · 2022-09-01 (published)kubernetesobservability
Grafana Labs was migrating one Grafana Cloud Logs cluster so that the ring, the distributed hash table Loki components use to find each other, would live in embedded memberlist instead of a separately deployed key-value store, to remove that store as a single point of failure. Pods that join memberlist carry the label gossip_ring_member: 'true'. Services are generated with jsonnet and Grafana Tanka, and the library function that built a Service for a Deployment or StatefulSet copied every label of the workload into the Service selector. Another team had already flagged this and added an optional parameter to leave some labels out, but the author did not notice it and did not pass it, so the distributor Service ended up selecting only pods that had the new label.
The gateway, which authenticates write requests and finds the distributors through the Kubernetes API, could therefore only see the new pods once the rollout began. 100 percent of write traffic went to about 10 percent of the distributors. Those pods ran past their CPU and memory limits, were OOM-killed and failed their readiness checks, so Kubernetes did not move on to replace more old pods, and the same few pods kept taking the whole load. Because distributors are the entry point for data, the write path of that cell was down for about 25 minutes.
Recovery started before the cause was found. Treating the outage as OOM crashes, the team scaled the distributor Deployment out. The extra replicas came up with the new label, joined the Service and spread the load; fewer pods crashed, more became healthy, and Kubernetes continued the rollout. The migration step was then reverted. The cause was found in the Kubernetes manifest diffs that automation posts on every pull request: the selector had been added to the distributor Service and the Service had been recreated, which a normal deploy does not do. The fix was to set service_ignored_labels:: [$._config.gossip_member_label] and pass it to the Service constructor k.util.serviceFor.
The step had not caused an outage elsewhere because this environment scaled its distributors horizontally much more than the others. Environments of similar size also gave distributors more memory, so the new pods survived the extra share of traffic long enough for Kubernetes to keep rolling the old pods.
During a rollout of the Grafana Loki distributor Deployment in one Grafana Cloud Logs environment, the write path went down for about 25 minutes. The newly started distributor pods were OOM-killed over and over, never passed their health and readiness checks, and the rollout stopped making progress.
kubectl get pods and the Kubernetes metric kube_deployment_status_replicas_updated showed that only 16 of 140 distributors, about 10 percent, were in the updated state for the whole outage. Counting the per-pod series of loki_request_duration_seconds_count whose rate on the write route was above zero gave the same answer: only 16 of 140 distributor replicas were receiving write requests.
The first suspicion was the data itself. Because log streams are sharded and replicated to subsets of pods, the team thought a single high-volume stream or a burst of new streams might have arrived just as the deploy started. The metrics for incoming write request rate and for active streams had not changed before the incident, so the suspicion moved to how requests were being routed to the distributors.
The same migration step had already been rolled out to other environments. It produced a few errors there, which the team took for the usual small number of errors during a rollout of many components that start at different times.
Symptom ? Service outageOOMKilledLoad imbalanceProbe failures
Root cause ? Configuration change
The first clue was kubectl get pods and the kube_deployment_status_replicas_updated metric showing that only 16 of 140 distributor pods had been updated.
A Service whose selector is built by copying every label of a workload changes which pods it matches as soon as a new label is added. During a rollout it then matches only the new pods, which receive all the traffic before there are enough of them.
A few errors during a rollout in other environments hid the problem, because those environments gave distributors more memory and could absorb the shift. Manifest diffs kept for every pull request made it possible to see that the Service selector had changed.
same root cause · shared Container memory limit
same root cause · shared Configuration templating
same root cause · shared Configuration templating
same root cause · shared Configuration templating
same root cause · shared Container memory limit