IL-0143 Google Cloud · 2019-11-04kubernetesgcp

GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image

From 4 November 2019 some GKE nodes running one of thirteen 1.13 and 1.14 patch versions had more kernel panics than normal after being upgraded. While a fix was prepared, Google advised affected users to go back to an earlier patch release not on the affected list, and users on a Release Channel to contact support to downgrade their nodes.

Fixed versions 1.13.11-gke.14, 1.13.12-gke.8, 1.14.7-gke.23 and 1.14.8-gke.12 became available on 11 November and were rolled out to node pools with auto-upgrade enabled; clusters on manual upgrades had to upgrade their nodes themselves. The fix downgraded the Container-Optimized OS image to cos-73-11647-293-0 as a temporary mitigation, and Google expected the next GKE release to bring an upgraded kernel with a fix for the panics. The incident was closed on 13 November at 15:38 US/Pacific.

Severity: –
Scope: Some nodes in GKE clusters upgraded to one of the affected 1.13 and 1.14 versions
Users affected: A very low number of projects
Region: –
Time to detect: –
Time to resolve: 9d 3h 52m

Symptoms

Some nodes in Google Kubernetes Engine clusters that had recently been upgraded showed an elevated number of kernel panics. The status page described it as kernel panics following an upgrade in GKE version.

The affected GKE versions were 1.13.11-gke.9, 1.14.7-gke.14, 1.13.12-gke.1, 1.14.8-gke.1, 1.13.11-gke.11, 1.13.12-gke.2, 1.14.7-gke.17, 1.14.8-gke.2, 1.13.12-gke.3, 1.14.8-gke.6, 1.13.11-gke.12, 1.13.12-gke.4 and 1.14.8-gke.7. On 7 November the incident was downgraded to an orange Service Disruption because very few projects were actually affected.

Symptom ? Node kernel panic

Root cause

Root cause ? Software defect

Component ?Node operating system image Linux kernel

Contributing factor ?–

Trigger: Upgrading GKE node pools to one of the affected 1.13 or 1.14 versions.

Detection

–

Remediation

advised affected users to downgrade to a patch release not on the affected list, with support helping Release Channel users
released GKE 1.13.11-gke.14, 1.13.12-gke.8, 1.14.7-gke.23 and 1.14.8-gke.12, which downgraded Container-Optimized OS to cos-73-11647-293-0
rolled the fixed versions out to node pools with auto-upgrade enabled; clusters on manual upgrades had to upgrade their nodes themselves

Action items

Ship a GKE release with an upgraded kernel that fixes the panics planned

Lessons

Downgrading the node operating system image was used as a temporary mitigation until a kernel fix was available.

Sources

https://status.cloud.google.com/incident/container-engine/19012 · official-status

References

Related cases

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022
same root cause · shared Node operating system image, Linux kernel
IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020
same root cause · shared Node operating system image, Linux kernel
IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025
same root cause · shared Node operating system image
IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025
same root cause · shared Node operating system image
IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025
same root cause · shared Linux kernel
IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNIMicrosoft Azure · 2020
same root cause · shared Node operating system image
IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020
same root cause · shared Linux kernel