IL-0063 AdGuard VPN · 2025-12-02kubernetes
AdGuard VPN runs a Redis cluster in Kubernetes as a StatefulSet. A StatefulSet declares its disks in volumeClaimTemplates, and that field cannot be changed after creation. When the Redis disks were running out of space, the team used the storage backend's online volume expansion: the disk size was raised in Git at 7:00 UTC, the PersistentVolumeClaims were resized by hand at 7:05, and by 7:10 every pod had its larger disk without a restart. That left the StatefulSet template in Git and the cluster's real volumes describing different sizes.
To remove that drift, at 8:10 UTC the team deleted the StatefulSet with orphan deletion, expecting the pods and disks to keep running untouched, and recreated it with the new template. Kubernetes did not know the change was already in place and reconciled the workload, which restarted the whole Redis cluster between 8:10 and 8:15. The Redis pods had no startup or readiness probes, so a pod that was merely running counted as ready and Kubernetes moved on to the next pod before Redis could serve its load. The cluster came back partly degraded, with replicas attached to stale masters.
From 8:15 to 8:35 UTC engineers repaired the cluster by hand: they made the masters forget obsolete node identities, reset the affected replicas, rejoined them and waited for replication to catch up. Meanwhile reconnect storms caused cascading out-of-memory failures in the backend service, dealt with from 8:35 to 8:50 by scaling the backend, raising memory limits for the affected component and temporarily relaxing probes so instances could take traffic immediately. The service was back, with slow responses, after about 50 minutes of downtime (8:10 to 9:00 UTC), and the temporary workarounds were reverted at 10:00 UTC.
The day began with an alert that several instances of the Redis cluster were running out of disk space. That part was fixed quickly. The outage came an hour later: right after the Redis StatefulSet was deleted and recreated at 8:10 UTC, Kubernetes restarted every Redis pod within about five minutes, and problems started to appear.
At first the damage did not look large. The backend service logged complaints about its Redis connections, but the overall service graphs did not collapse straight away. When the restarts settled, the Redis cluster was partly degraded: some replicas could not rejoin because they still held stale information about which master they belonged to.
While those replicas were being repaired and resynced, the backend service was hit by reconnect storms and higher load and began failing with out-of-memory errors, one after another. The team at first blamed the wrong component for the OOM failures. Scaling the backend was slow because there was little spare capacity. Readiness probes made it worse: a pod marked ready was flooded with traffic, fell back to not ready, and pushed its load onto the next pod, which then failed the same way.
Root cause ? Configuration change
An alert about Redis instances running out of disk space started the work; the outage itself showed up as backend errors about Redis connections right after the StatefulSet was recreated.
The disk resize was not the problem; the cleanup afterwards was. An immutable field pushes operators toward deleting and recreating the controller, and that can set off a full reconcile. Startup and readiness probes are what slow a rolling restart down to a safe pace; without them a distributed store restarts in the most fragile way. Recovery plans must expect a cluster to heal slowly, and dependent services need to degrade gracefully while it does.
same root cause · shared Container memory limit
same root cause · shared Persistent volumes and stateful sets
same root cause · shared Container memory limit