IL-0032 Airbnb · 2020-12-04 (published)kubernetes

Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods

Airbnb's compute infrastructure team had rolled out a mutating admission controller within the previous month. When it needed to delete two items from an array in a spec, it generated two remove operations in ascending order, index 2 and then index 3. After the first removal the array no longer had an index 3, so the patch failed. The correct patch removes the higher index first. The bug was already well known once the team searched for the error message.

Because the patch failed, some deployments could no longer create new pods. The change had gone in seven days earlier, and during that time some deployments had not actually shipped new code: services kept running on their old replica sets, and as those older pods died one by one the services degraded slowly until one of them reached two pods and started failing. Deleting the mutating admission controller from the cluster stopped the outage immediately, and new pods came up.

In the postmortem the team found that metrics which could have revealed the problem earlier already existed, but no alert was built on them.

Severity: –
Scope: one production service failing and other deployments in the cluster silently not deploying new code for seven days
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

A production service at Airbnb showed a high error rate and could not be deployed. The deploy failed with a cryptic Kubernetes error about being unable to access invalid index 5.

The service had only two pods running while its horizontal pod autoscaler wanted nine replicas, and the service was struggling on what was left. The namespace held one deployment but three replica sets, and the replica sets that still had pods were 16 days old.

For about an hour the responders worked on the autoscaling and replica set state: they deleted the horizontal pod autoscaler and scaled up by hand, deleted the old replica sets while keeping the one with the last two pods, and created a fresh deployment object to force an update. None of this helped. Recreating the namespace was ruled out because it would have deleted the last two production pods. A second engineer was paged, and about an hour later the team went back to the error message from the first page, searched for it, and found that it came from JSON patching.

Symptom ? High error rate Throughput degradation Deploys failing

Root cause

Root cause ? Software defect

Component ?Admission webhook Horizontal pod autoscaler

Contributing factor ?Change with a delayed effect Monitoring blind spot Ignored early signal

Trigger: A change to the mutating admission controller, which had been rolled out within the previous month, had gone in seven days before the incident surfaced; when it had to delete two items from an array in a spec, it generated an invalid JSON patch that removed index 2 and then index 3.

Detection

The affected service reported a high error rate and its deploy failed; the slow degradation of other services over the previous seven days had gone unnoticed.

Remediation

deleted the mutating admission controller from the affected cluster

Action items

Create alerts from the existing metrics that would have shown the problem earlier unknown

Lessons

Keep in mind that a mutating admission controller exists in the cluster; a recently added one is easy to forget during an incident. Search for the exact error message early instead of working on the visible symptoms. Page a teammate when progress stalls.

Sources

https://www.youtube.com/watch?v=4CT0cI62YHk · talk

Related cases

IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)Wiz · 2025
same root cause · shared Admission webhook