IL-0064 Moonlight · 2019-01-18kubernetesgcp
Moonlight's website and API had intermittent outages from January 18 to January 22, ending in periods of complete downtime. The application ran on a managed Kubernetes cluster on Google Cloud with three nodes, and every web-facing application had three replicas. The team expected one replica per node, so that two nodes could fail before the site went down.
The scheduler did not spread the replicas. It sometimes put all three website pods on one node, together with other CPU-heavy work such as server-side rendering. Those pods drove the node to 100% CPU, the node's kernel panicked, and the node stopped responding. The scheduler then moved all the crashed pods to another node, which went through the same cycle, and the effects compounded. What started with the Redis pod ended with every web-serving pod offline.
The team first assumed a Google Cloud network outage, because Google had reported networking problems at the time. Google Cloud Support identified the link between sustained full CPU and kernel panics. Adding anti-affinity rules to all major deployments, so their pods are spread across nodes, restored the site. The cause of the kernel panics themselves was still being investigated with Google when the post was written.
On Friday, January 18, Moonlight's API began returning occasional errors that looked like connection problems with its Redis database. The API checks Redis on every authenticated request to validate the session. At the same time the Kubernetes monitoring reported some nodes and pods as unresponsive.
Over the weekend, as traffic dropped, the errors mostly went away. On Tuesday morning the website was completely offline, with no external traffic reaching the cluster at all.
The team first blamed Google Cloud: Google had reported networking service disruptions, and another person on Twitter described similar symptoms. Google Cloud Support escalated the case to its support engineers, who found the real pattern: whenever a node ran at 100% CPU for a sustained period, the virtual machine hit a kernel panic and crashed.
The failure grew over the days. At first only the Redis pod was crashing; later every pod serving web traffic went offline. Exponential back-off rules made the periods of downtime longer.
Symptom ? High error rateIntermittent connection failuresNode NotReadyService outageSustained high node CPU
Root cause ? Capacity exhaustion
API errors that looked like Redis connectivity problems, and Kubernetes monitoring that showed nodes and pods as unresponsive. Google Cloud Support later found the link between sustained 100% CPU and kernel panics.
Three replicas give no fault tolerance if the scheduler places them all on one node; anti-affinity rules make the spread explicit. A node failure that comes from the workload itself moves with the workload when the scheduler reschedules it.
same root cause · shared Pod scheduling and placement, Application database
same root cause · shared Pod scheduling and placement
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Application database
same root cause · shared Linux kernel
same root cause · shared Application database
same root cause · shared Pod scheduling and placement