IL-0067 BC Government · 2025-07-09kubernetes

Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart

Klamm runs on OpenShift with a Redis cluster as its cache. On 9 July 2025 a Laravel worker produced a large I/O spike, which led to a surge of requests to Redis. Redis's append-only file, the log it writes every change to so it can rebuild its data after a restart, grew quickly and filled the persistent volume of the Redis master, whose storage limit request was 50Mi.

When fsync of the AOF failed for lack of space, Redis could not write its buffered changes to disk and the process stopped, so the container crashed. The last writes to the AOF had not completed cleanly, which left the file corrupt. On each restart Redis tried to load the AOF to recover its data, could not parse it, and crashed again. With the cache down, queue workers timed out and the application went down.

The team backed off the StatefulSet and the deployment, deleted the corrupted PVC and rebuilt it, and restarted both. Rebuilding the PVC lost the data on it. OpenShift events from the incident were not retained.

Severity: sev1
Scope: Klamm production application and its dependent services; data on the Redis volume lost when the PVC was rebuilt
Users affected: –
Region: –
Time to detect: –
Time to resolve: 30m

Symptoms

The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.

The Redis master's logs showed "Fail to fsync the AOF file: No space left on device". After the Redis container stopped, every restart failed while loading the append-only file (AOF) with an "invalid byte sequence" error, and the pod kept crashing without becoming ready.

Symptom ? Service outage Container crash loop Persistent volume full

Root cause

Root cause ? Capacity exhaustion

Component ?Persistent volumes and stateful sets In-memory cache

Contributing factor ?No capacity planning

Trigger: A large, unexpected spike in I/O from a Klamm Laravel worker sharply increased requests to the Redis cache.

Detection

–

Remediation

backed off the Redis StatefulSet and the deployment
deleted the corrupted PVC and rebuilt it
restarted the StatefulSet and the deployment

Action items

Increase the Klamm Redis cache volume to 100Mi planned
Integrate Sysdig monitoring on the namespace planned
Enable replica pods on the Redis cluster planned

Lessons

–

Sources

https://github.com/bcgov/klamm/wiki/Incident-Post%E2%80%90Mortem-Template · github

Related cases

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025
same root cause · shared In-memory cache
IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hoursMagic Pages · 2025
same root cause · shared Persistent volumes and stateful sets
IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1
same root cause · shared In-memory cache