IL-0034 Airbnb · 2019-11-22 (published)kubernetesobservability
This case comes from a KubeCon talk by two Airbnb engineers about ten ways Kubernetes broke at Airbnb. Airbnb's service discovery system, SmartStack, puts an HAProxy sidecar in every pod as its outbound proxy and decides which pods receive traffic. Its logic started from "pod ready = true", went through every container in the pod, set the pod to not ready if any container was not ready, and then published only ready pods to receive traffic.
Health checking in Kubernetes is per container, so every sidecar counts toward the pod's readiness. Earlier, to stop a failing stats agent from affecting services, the team had left probes off non-critical containers. When the stats agent crashed during a planned upgrade, the crashed container was not ready anyway, the service mesh logic marked the whole pod not ready, and all pods in the cluster stopped receiving traffic.
The fix went into the service mesh: containers such as the logger, tracing and the stats agent are labelled non-critical, and one added line in the readiness loop skips them. The speakers call it a stopgap that is still changing.
An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.
The stats agent runs as a sidecar container in the pods. During the upgrade it crashed, and the crashing container made each pod unready. Because Airbnb's service discovery only sent traffic to pods it saw as ready, every pod in the cluster dropped out of traffic.
The team had expected this could not happen: they had already decided that a failing stats agent must not take a service down, and had removed the probes from non-critical containers such as logging, tracing and metrics. Removing the probes did not help, because a container that crashes fails its readiness check whether or not it has a probe.
Symptom ? Probe failuresService outage
Root cause ? Faulty deployment
A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.
–
Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent