IL-0172 Devsisters · 2021-02-19kubernetesaws
Devsisters ran the CockroachDB cluster for Cookie Run: Kingdom on Kubernetes from one Helm chart. CockroachDB's Locality feature spreads replicas so that losing a whole zone or region does not lose data, but an operator has to set the topology, and the availability zone of a pod is only known after it is scheduled and is not available through the Kubernetes API. Because working around that would add a lot of complexity, and because Devsisters had not seen an AZ failure on AWS since 2013, the team prepared for the more common case of several instances being retired at once: the replication factor was set to 7 so that three nodes could fail at the same time without data loss.
On 19 February 2021 a cooling failure in an AWS Tokyo data centre took down 6 of the 60 nodes. The chance that four or more of a range's seven replicas sat on those six nodes was about 0.09% per range; across 25,000 ranges about 23 were expected to be hit, and 2 data ranges and 32 system ranges were lost. Cockroach Labs support said the system ranges could be recovered, so the team concentrated on the two data ranges holding user data.
Since a backup from the surviving 54 nodes failed, the team copied data outside the broken ranges into new tables with statements of the form INSERT INTO new_table SELECT * FROM old_table WHERE {unaffected_ranges}. Restoring from periodic backups was too slow, with no end in sight after 8 hours. In the CockroachDB source the team found the undocumented command debug unsafe-remove-dead-replicas, which tries to recover ranges that lost quorum; run on a screen share with Cockroach Labs engineers, it saved one of the two ranges. The last range, which could affect an estimated 500 to 2,000 users, was restored with the bit-level storage recovery method used in the team's previous incident, and in the end no user data was lost.
During a team dinner, a Slack alert reported that one node of the CockroachDB cluster behind Cookie Run: Kingdom had gone down. The DevOps engineers were not worried at first, because CockroachDB is built to survive the loss of some nodes.
Then more nodes went down: two more three minutes later, two after six minutes, two after twenty minutes, until 6 of the 60 nodes were out within 22 minutes, all with a host status failure. (One sentence of the write-up gives the window as 12 minutes.) Community sites and other known AWS customers showed the problem was not limited to Devsisters, and the cause turned out to be a failed cooling system in an AWS Tokyo region data centre that overheated the servers.
With six nodes down, more than the three the cluster had been sized to survive, 2 data ranges and 32 system ranges out of about 25,000 ranges lost quorum. A table backup taken from the 54 surviving nodes did not work, which the team suspected was because of the broken ranges.
Symptom ? Node NotReadyData lossService outage
Root cause ? Cloud provider outage
A Slack alert reported that a database node had gone down.
A replication factor sized for random instance retirements does not protect against a whole availability zone failing; for a game of this scale a multi-AZ placement is required, not optional.
Keeping one uniform Helm chart made zone topology hard to set, which is part of why it was left out. A repetitive chart with a values file per zone sets it reliably.
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Application database