IL-0061 Spotify · 2019-05-22 (published)kubernetesgcp
At the time Spotify ran three production GKE clusters, one each in the US, Europe and Asia, backed up every hour. Stateless backend services were moving to GKE only partially: teams were told to put part of each service's capacity on Kubernetes and keep the rest on ordinary VMs. After an engineer deleted a 50-node production cluster in November 2018 by working in the wrong browser tab, the team started managing clusters with Terraform, from a git repository that both the cluster operators and the feature teams commit to.
The team did not yet understand that Terraform decides what to do from its state file. Their pull request to import the Asia cluster ran review builds that modified the global state file used for production, before the pull request was merged. A few minutes later a feature team opened an innocuous-looking pull request adding three users to an RBAC binding; what mattered was what it lacked, the definition of the Asia cluster. That pull request was merged first, Terraform found a cluster in state that was not in the configuration, and deleted it.
Merging the import pull request to recreate the cluster failed halfway: Terraform now ran as a service account that lacked permissions the engineers' personal owner accounts had had. The service account was given the permissions it needed, but these differed from those used for the import, and with different permissions the Google APIs return different attributes for a cluster. Terraform saw the clusters as changed and deleted another production cluster. Recovery from this second deletion lasted from 8 in the evening until 5 in the morning.
Listeners were not affected because of choices made for the migration. Services ran only partly on Kubernetes, and service discovery registered pod IPs directly. After the operators removed a deleted cluster from the service discovery configuration and restarted it, the pod IPs dropped out and clients moved to the instances outside Kubernetes. The hourly cluster backups, whose restore had already been tested, were essential to the recovery.
Terraform, applied from the shared cluster configuration repository after a merge, deleted Spotify's production GKE cluster in Asia. While the team tried to recreate it, Terraform deleted a second production cluster, so two of the three production clusters were gone.
The internal service discovery system polls pod IPs from each cluster; when it could not reach a cluster, the cluster operator team was paged. Inside the company, one team had to create more VMs outside Kubernetes to carry the load, the operators had to find and update every place where the old clusters' master IPs were hardcoded, and everyone using kubectl had to refresh cluster credentials because the recreated clusters had new certificates. Otherwise the internal user support Slack channel stayed quiet, and there were no reports of impact on listeners.
Symptom ? Resources deleted
Root cause ? Configuration change
Internal service discovery could not reach the deleted cluster and the cluster operator team was paged.
Plan for failure from the start and migrate large infrastructure gradually, so there is redundancy and a way back. A backup whose restore has never been tested is not a backup.
Practising disaster recovery paid off: restoring the first deleted cluster took three and a quarter hours, and the team can now restore a larger cluster in one hour. What matters is not who made the mistake but what was learned from it.
same root cause · shared Terraform
same root cause · shared Terraform
same root cause · shared Terraform
same root cause · shared GKE control plane