IL-0196 Palark · 2020-03-13 (published)kubernetes
Palark runs Redis for a customer's large website on Kubernetes with the Redis Operator by Spotahome, which builds a Redis StatefulSet, a Sentinel Deployment and ConfigMaps from a RedisFailover custom resource. One installation was used as a persistent store without TTLs; its .rdb snapshot was over 5 GB and the single Redis pod used about 16 GB of RAM at peak while saving to disk, so at first only one replica fit on the nodes.
After new servers were added, the team raised the replica count to two to make the database more reliable and, in the same change, cut the memory requests and limits from 22 GB to 19 GB so the pod could be rescheduled more easily. The resource change forced a rolling update of the StatefulSet. Because the operator's readiness probe was a plain ping, the new replica counted as ready before it held any data, the rolling update restarted the only pod with data, Sentinel promoted the empty replica, and replication from that empty master erased the original data. The result was downtime and a restore from backup.
The author names the operator's readiness probe as the principal cause and opened an issue with the operator project; the developers answered with a fix, which Palark tested. The author also points out that with this operator a single-replica setup was safer against data loss than a multi-replica one, because a master restart while a new replica is joining loses the data. A later analysis of the dump with redis-memory-analyzer found 27,402,900 keys, 16,236,021 of them (59.24%) under one key pattern, and the developers found over 18 million excess keys, which they attributed to a wrong architectural decision in the application.
Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.
The sequence, as reconstructed afterwards: the StatefulSet grew to two pods and the new pod joined as a slave and began replicating. The operator's readiness probe, which only sends a plain ping, marked the new pod ready almost at once. Because the memory requests and limits had also changed, the StatefulSet rolling update then restarted the one pod that held the data.
The Sentinels saw the master disappear and promoted the new, still empty replica to master. At that moment the old master had hardly managed the initial BGSAVE needed for a full resynchronisation. When the rolling update finished, the restarted original pod connected as a replica of the empty master and synchronised from it immediately, which wiped its data.
Symptom ? Data lossService outage
Root cause ? Software defect
–
Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty.
A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Container memory limit
same root cause · shared Persistent volumes and stateful sets