IL-0193 Grafana Labs · 2019-12-07kubernetesgcpobservability
Grafana Cloud's hosted Graphite service runs Metrictank on Kubernetes, and in the US-East cluster Metrictank stores data in Cassandra, also on Kubernetes as a StatefulSet. On December 7, a GCP SSD persistent disk incident caused kernel deadlocks and filesystem corruption on the nodes, the nodes rebooted, and some could not reattach Cassandra's volumes afterwards. All Cassandra pods failed. The StatefulSet used the OrderedReady pod management policy, so Kubernetes started the pods one at a time, and the second pod's volume was still unavailable after GCP said most volumes were working, which blocked every pod after it. The team deleted and recreated the StatefulSet with a different pod management policy; all pods but one started, which was enough for Cassandra to work.
Metrictank then could not reach the recovered cluster. It discovers Cassandra node IPs through the nodes it is already connected to, and because every pod had a new IP it had no connection left and never re-resolved the service hostname; this failure mode had never been considered. All Metrictank instances had to be restarted, which took more than 20 hours. Kafka retention was raised so no data was lost, and extra nodes were added so the restarts would not be starved of CPU.
While replaying the larger-than-usual backlog from Kafka, some Metrictank instances hit an old deadlock: when the write queue to Cassandra is full the ingester blocks while holding an internal lock, and the callback that runs after a successful write needs the same lock, so the queue can never drain. The bug had appeared once or twice over many years and had been worked around by raising the queue size. Customers could not run these queries for 6 to 23 hours.
An alert fired because some Metrictank instances could not drain their write queue, which meant they could not write to Cassandra. Google had not yet published an incident, but a team member's Discord server, also hosted on GCP, was down, which pointed at GCP; Google confirmed an SSD persistent disk incident about 90 minutes after it began.
On the compute nodes the degraded SSD volumes caused kernel deadlocks and mild filesystem corruption, and node monitoring processes rebooted the nodes. After reboot some nodes could not attach the volumes the Cassandra pods needed, which led to more automatic reboots. Writes to Cassandra either failed or took more than 2 seconds. Adding compute nodes, in the hope that new nodes could attach the disks, did not help.
For customers, Graphite queries that needed data not held in Metrictank's in-memory cache failed; recent hours of data are always served from memory, so the error rate rose slowly and peaked at 20% of queries. Once GCP recovered, the Cassandra StatefulSet still did not come back: with the OrderedReady pod management policy each pod waits for the previous one to become healthy, and the second pod's volume was still unavailable, so no later pod started.
After Cassandra was running again, no Metrictank instance could reconnect to it. Every Cassandra pod had been recreated with a new IP address, and Metrictank, which learns new node IPs only from nodes it is still connected to, did not resolve the service hostname again.
Symptom ? High error rateNode NotReadyPods PendingService outage
Root cause ? Cloud provider outage
An alert that Metrictank write instances could not flush their write queue to Cassandra. Error rates alone rose too slowly to notice quickly, because most queries were served from memory.
An OrderedReady StatefulSet cannot recover past a single pod whose volume will not attach; when every pod is down, the ordering that protects normal start-up blocks the whole cluster. A client that learns peer addresses only from live peers needs a way to go back to the service name when all peers are replaced at once.
A long-tolerated bug with a config workaround can surface at the worst time; this one turned out to have a simple fix. Writing incoming data to a Kafka write-ahead log first meant all data could still be ingested after recovery, and Kafka on non-SSD disks escaped the disk incident. With both Metrictank bugs fixed, the team expects a similar event to take less than 3 hours to recover from instead of 23.
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Application database