IL-0036 Universe · 2017-07-21kubernetes

A reindex Job that kept failing exhausted one node after another until the whole cluster was down

Universe had recently moved this service to Kubernetes and ran background tasks as Kubernetes Jobs. To fix poor search results in Discover, the team redefined a search index to exclude some fields from full-text search and started a reindex job.

The Job's pod consumed all resources on its node, the node went unhealthy and was killed, and Kubernetes started the Job again on another node. The team had set RestartPolicy=Never and expected that to stop retries, but that setting applies to the pod, not to the Job: a Job starts new pods after a failure and may run more than once. The Job therefore moved from node to node and took each one down, until all nodes were under repair and new nodes were killed as soon as they joined.

The outage ended when the team removed the Job from the scheduler; the nodes then auto-healed back to a healthy state. The status page shows the investigation opened at 14:37 EDT and the incident resolved at 14:55 EDT.

Severity: –
Scope: the reporting and search APIs and the Kubernetes cluster UI; every node in the cluster
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

Nodes failed in sequence. The job used up all available resources on the node it ran on, that node became unhealthy and was killed, and the job was started again on another node, which then failed the same way. This went on until every node was being repaired, and each replacement node was killed soon after it came online.

Symptom ? Service outage Node NotReady Node recreation loop

Root cause

Root cause ? Capacity exhaustion

Component ?Jobs and CronJobs Pod scheduling and placement

Contributing factor ?Unverified assumption Automation amplification Excessive blast radius

Trigger: A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

Detection

The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

Remediation

removed the reindex Job from the scheduler
let the nodes auto-heal back to healthy

Action items

–

Lessons

restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

Sources

https://status.universe.com/incidents/115n3vxqwzcf · official-status

References

https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#handling-pod-and-container-failures

Related cases

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025
same root cause · shared Pod scheduling and placement
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024
same root cause · shared Pod scheduling and placement
IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019
same root cause · shared Pod scheduling and placement