IL-0190 Grafana Labs · 2020-03-16kubernetesobservabilitynetworking
Grafana Cloud Hosted Prometheus is built on Cortex. To deduplicate samples from pairs of HA Prometheus replicas, the Cortex distributor records in etcd which replica is currently accepted; if no samples arrive from that replica for 15 seconds, the next replica that writes becomes the accepted one. On March 16 a scheduled Kubernetes upgrade terminated the node running the etcd leader. etcd elected a new leader within about 10 seconds, but two distributor replicas kept a TCP connection open to the old leader, because the node died without closing it. Their etcd reads timed out and the writes passing through them failed.
The etcd client in Cortex supports gRPC keepalive probes, but Cortex had not configured them, so the client never noticed the dead connection. The team reproduced the failure in dev by dropping packets with iptables between a distributor and its etcd instance, and showed that setting DialKeepAliveTime, DialKeepAliveTimeout and PermitWithoutStream in the clientv3 configuration prevents it.
The upgrade scripts also played a part: the etcd pods are managed by the etcd operator, so the scripts did not know how to move them gracefully, and the API call the scripts used to terminate an instance gave it only about 90 seconds to shut down. Prometheus remote_write treats the timeout as recoverable, so senders retried the failed writes against other distributor replicas and no data was lost; storage was only delayed.
About a minute after the etcd leader changed, two replicas of the Cortex distributor started logging "context deadline exceeded" when they tried to read keys from etcd. Writes from some customers that landed on those two replicas failed, while the other distributor replicas kept working, so the service saw an error rate of about 20% on writes to the London region of Hosted Prometheus.
The distributor reads etcd to know which replica of a customer's HA Prometheus pair is currently elected to have its writes accepted. Another etcd replica had become the leader about 10 seconds after the old one disappeared, but the two distributors were still holding TCP connections to the old leader, whose node had been terminated without closing them.
Symptom ? High error rateIntermittent timeouts
Root cause ? Software defect
An SLO-based alert paged the on-call engineer at 11:31, 11 minutes after the errors started. At a 20% error rate the monthly SLA would have taken about 18 hours to breach.
When a node dies abruptly, clients holding TCP connections to it are not told; without keepalive probes an etcd client can keep sending requests into a dead connection long after a new leader exists. A failover that works on the server side still needs clients that detect the dead peer.
Node replacement during cluster upgrades should drain pods gracefully, including pods owned by operators that the upgrade scripts do not know about, and should give instances enough time to shut down.
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared etcd
same root cause · shared Node pool
same root cause · shared etcd
same root cause · shared etcd