IL-0202 Honeycomb · 2023-11-05kubernetes
On November 5, 2023, Honeycomb's US1 event ingestion was partly available for a little over 1 hour 10 minutes and completely down for about 5 minutes. A sudden, very large number of queries from many separate connections reached the ingest hosts before auto-scaling could react; Honeycomb suspects recent optimisations to its ingest code, which shifted the workload somewhat, slowed auto-scaling down. The requests set off many database writes that interfered with each other and clogged the connection pools. This happened faster than the cache, which would have avoided that work, could spread the writes to all hosts, and it drove up memory use on the ingest hosts until they died.
Load shedding did not prevent the restarts. When engineers scaled up the fleet, Kubernetes' crash loop back-off kept the previously failing hosts offline, so capacity stayed too low. Engineers cut off most ingest for a few minutes to build back enough ingest hosts, which recovered the system quickly. By the time the hosts were back, the writes had gone through and the caches had refreshed themselves, but some customers were still sending more than 20 times their usual traffic, probably whole-batch OTel retries. Honeycomb temporarily lowered the rate limits for those sources until the traffic subsided, and the pipeline became stable.
Honeycomb describes this as a variation of earlier incidents for which it already had mitigations, but this time the failure got around some of them.
Standard alerting paged Honeycomb's on-call engineers about the performance and stability of event ingestion, and the automated load-shedding mechanisms tripped and sent further notifications. The status page described a higher than normal error rate and response time on the ingestion service.
From about 02:15 UTC, customers sending events to the API ingestion endpoint saw processing slow down and fail on and off, until ingestion stopped completely for a few minutes. The system came back briefly at 02:50 UTC but was not fully stable until 03:30 UTC. The incident severity was raised to critical as more ingestion traffic was dropped.
Load shedding was meant to drop traffic early enough to stop ingest hosts from failing one after another, but the ingest fleet still went through a series of restarts. Engineers scaled the fleet up by hand to add capacity, and then saw that Kubernetes' crash loop back-off was holding the hosts that had already failed offline, so the cluster still did not have enough capacity.
Some traffic sources retried so aggressively that it looked like a thundering herd, and some customers were sending more than 20 times their usual traffic. Honeycomb first took this for a backfill, then suspected retries: the OTel protocol can only report success or failure for a whole batch of spans, so when Honeycomb dropped part of a batch, clients likely re-sent the entire batch.
Symptom ? High error rateHigh latencyContainer crash loopService outageDatabase overload
Root cause ? Capacity exhaustion
Standard alerting notified the on-call engineers about ingestion performance and stability, and the automated load-shedding mechanisms added notifications when they tripped. The first status post, about alerts on a higher than normal error rate and response time, went up at 18:28 PST (02:28 UTC), 13 minutes after the 02:15 UTC start.
Load shedding and auto-scaling were in place, yet a burst that arrived faster than the cache could spread its writes still took the ingest hosts down. During recovery, Kubernetes' crash loop back-off worked against the operators: the hosts that had failed stayed offline while the fleet needed every host. Clients that retry whole OTel batches after a partial drop can multiply traffic many times over, so per-source rate limits were what finally stabilised ingest.
same root cause · shared In-memory cache, Application database
same root cause · shared In-memory cache
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Application database