IL-0197 Palark · 2021-05-27 (published)kubernetes
Palark was upgrading a non-critical part of its internal production infrastructure, used for testing technical solutions, from Kubernetes 1.15. Its persistent volumes came from the Rook v0.9 operator, whose Helm release contained resources with deprecated API versions, so instead of upgrading Rook in the running cluster the team moved data by hand to storage classes not managed by Rook. One of the workloads was a three-node Elasticsearch cluster deployed as a StatefulSet from a Helm chart.
The plan was to change the storage class in the chart's volumeClaimTemplates, delete the StatefulSet without cascading, redeploy it with six replicas, drain all shards from the three old nodes, and then recreate replicas 0, 1 and 2 one by one on new PVCs. The chart's StatefulSet template, not the reviewed ConfigMap, also set the environment variable cluster.initial_master_nodes to the name of pod 0. That setting decides the master-eligible nodes when a cluster starts for the first time. When elasticsearch-0 started with an empty volume, it began bootstrapping a new cluster and ignored the existing master in elasticsearch-2, leaving two clusters that both received writes.
The team first scaled the three deployments of workers that write to Elasticsearch down to zero, accepting daytime downtime over growing data loss. It then listed elasticsearch-0, -1 and -2 under initial_master_nodes in the ConfigMap and removed the environment variable from the StatefulSet, which started a rolling update. Unsure whether master election would be safe during a fast rolling update, the author saved and deleted the elasticsearch discovery Service and deleted the PVC of pod 0, which left the nodes unable to see each other (`Open Distro Security not initialized.`). After setting initial_master_nodes to elasticsearch-3, -4 and -5, creating an empty PVC for pod 0, deleting all six pods and restoring the Service, the six nodes formed one cluster and no index was in a state other than green. The author later found the election would have gone fine without these extra steps.
After the last old Elasticsearch pod, elasticsearch-0, was deleted and came back on a new, empty persistent volume, `_cat/nodes` run inside that pod listed a single node: elasticsearch-0 itself, marked as master. The same command run inside elasticsearch-1 listed the other five nodes, elasticsearch-1 to elasticsearch-5, with elasticsearch-2 as master.
The Elasticsearch cluster had split into two separate clusters, and new data was being written at random to one or the other. This happened in the middle of a planned migration of the three-node cluster in the kibana-production namespace from volumes of the rbd StorageClass to the high-speed StorageClass, done without downtime. The StatefulSet had been deleted with --cascade=false, redeployed with the new storage class and scaled to six replicas, shards had been moved off the three old nodes with cluster.routing.allocation.exclude._ip, and then the pods and PVCs of replicas 2, 1 and 0 were deleted one at a time. Replicas 2 and 1 rejoined the cluster on their new volumes; replica 0 did not.
The ConfigMap had been checked carefully before the migration and still looked correct. Taking elasticsearch-0 out of the Service by removing its label was considered as a way to stop writes to it, and rejected: the members find each other through the same Service (the DISCOVERY_SERVICE variable feeds discovery.zen.ping.unicast.hosts), so the pod could not have rejoined the cluster afterwards.
Symptom ? Split brain
Root cause ? Operator error
The engineer running the migration checked `_cat/nodes` after elasticsearch-0 restarted and saw it alone in a cluster of its own.
A setting that only matters when a cluster first starts, such as Elasticsearch's cluster.initial_master_nodes, can sit in a Helm chart template rather than in the ConfigMap that was reviewed. When a pod named in it starts with an empty data directory, it can bootstrap a new cluster beside the existing one.
The author's conclusion is that a migration plan like this one breaks on any surprise and needs a plan B ready. Before running it in a more important environment, rehearse it in a test environment configured like production, and either schedule downtime or move the load to another cluster: stop the writers, take a fresh backup, then move the data.
same root cause · shared Persistent volumes and stateful sets