IL-0100 PostHog · 2025-10-21kubernetesaws
Between 21 and 30 October 2025 the PostHog feature flags service had four incidents with more than 14 hours of major impact in total. Three of them had the same root cause: the Kubernetes nodes were too small relative to the pods' resource requests, so Kubernetes packed too many flags pods onto each node and node CPU went above 90%. The fourth, on 24 October, was a separate rate limiting misconfiguration: the tower-governor middleware by default does not trust X-Forwarded-For, so it saw all traffic as coming from the load balancer's IP, and about 97% of requests got 429 errors for 72 minutes until rate limiting was disabled. The rate limiting had been tested only directly against the service, not behind the production load balancers.
Under that CPU pressure, degraded requests made Envoy retry between the load balancer and the service. Each retry created new concurrent requests, each request did several concurrent Redis reads, and on a cache miss the service synchronously loaded the full flag and team state from Postgres and wrote it into Redis. One slow request could therefore fan out into dozens of Redis operations, and the result was bursts of writes that overloaded Redis. At the same time every pod keeps its own Postgres connection pool, and building one needs TLS handshakes, authentication and connection setup, which is expensive on a CPU-bound pod. New pods could not finish this within the 20-second startup timeout and went into crash loops, which reduced healthy capacity further. Because the flags service shares its Redis instance with the main PostHog application, the Redis overload spread to the main app.
On 21 October the triggering deploy was rolled back within 2 minutes, but the data transfer and resource exhaustion continued; the service recovered at 23:28 UTC after connections and Redis writes were throttled (103 minutes). On 28 October the team stopped the rollout, scaled the US pods to zero until Redis recovered from the write storm, and brought them back gradually; full service returned at 21:31 UTC (123 minutes). On 29 to 30 October CPU saturation directly slowed queries for 7 hours 9 minutes. What fixed all three was increasing the pods' resource requests and adjusting the fleet size so fewer, better-resourced pods ran on each node, which brought per-node CPU down. PostHog also added CPU alerts, which had been missing throughout, limited Envoy retries, and configured tower-governor to trust X-Forwarded-For from its load balancer.
On 21 October 2025 at 21:45 UTC a change to timeout handling was deployed to the feature flags service, and two minutes later automated monitoring reported higher error rates. The change was rolled back by 21:49, but error rates stayed high: about 38% of flag evaluation requests in the US datacenter returned errors. At 22:30 Redis metrics showed memory exhaustion on ElastiCache, at 22:35 Postgres connections spiked, and at 22:45 the team found a massive data transfer from Postgres into Redis. The main PostHog application was affected as well, because it shares the Redis instance with the flags service.
On 24 October IP-based rate limiting was deployed to the /flags endpoint at 18:00 UTC, and from 18:01 about 97% of evaluation requests worldwide received 429 (rate limit) errors. No alert existed for 429 errors, so the team learned of it from customer reports, with a 62-minute detection delay; the service recovered at 19:12 when rate limiting was disabled.
On 28 October a routine deployment with no flags code changes rolled the feature flag pods in the US region. From 19:15 the new pods could not connect to Postgres within the 20-second startup timeout and went into crash loops. Redis writes spiked from 19:20, Redis began evicting keys at 19:25 so the cache was effectively gone, the main PostHog app began to suffer at 19:26, and about 34% of US requests failed. On-call was paged by a high error count alert at 19:23.
From 22:30 UTC on 29 October to 05:39 UTC on 30 October queries from the flags service to Redis and Postgres were slow even though the metrics of both showed them healthy. Response times were several times the usual. Node CPU was above 90%, but no CPU alert existed, so this went unnoticed as the cause for more than 7 hours. Rolling back hardware changes at 00:03 removed most of the errors, but the latency stayed.
Customers saw errors and slow responses for all flag types, including read-only evaluations. On 28 October the team first tried the same remediation as on 21 October before finding other ways to reduce parallelism.
Symptom ? Sustained high node CPUHigh error rateHigh latencyContainer crash loopDatabase overload
Root cause ? Capacity exhaustion
On 21 October automated monitoring detected higher error rates two minutes after the deploy. On 24 October no alert existed for 429 errors, and the team learned of the incident from customer reports that began at 18:10 UTC, a 62-minute detection delay. On 28 October on-call received a high error count alert. On 29 October the incident was reported with higher error rates and latency. No alert existed for CPU pressure, so the CPU saturation behind all three incidents was not seen directly.
Nodes that are small relative to pod requests let the scheduler pack many pods on each node, and the resulting CPU saturation showed up three different ways: excessive parallelism, connection pool exhaustion and slow queries. Treating each symptom separately let the problem return; right-sizing the pods and nodes fixed all three.
CPU must be alerted on from the start. Unbounded Envoy retries and unbounded cache population amplified the overload, and a Redis instance shared with the main application spread a flags problem to the main application. A rollback did not end the incident, because the data transfer and resource exhaustion it had started kept running.
same root cause · shared Application database, In-memory cache
same root cause · shared Application database, Pod scheduling and placement
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared In-memory cache
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Pod scheduling and placement
same root cause · shared Application database
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Application database
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Application database
same root cause · shared Pod scheduling and placement