IL-0164 Linode · 2022-06-15kubernetes

An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silently

At about 18:30 UTC on June 15, 2022, Linode pushed an incremental update to LKE production that lacked changes from the previous release. As a result, cluster upgrades and node additions failed silently for customers. Linode added the missing changes to a follow-up update, released at about 21:00 UTC, which resolved the problem, and put additional steps and automation for updates in place.

Severity: –
Scope: LKE customers trying to upgrade clusters or add nodes
Users affected: –
Region: LKE in US-East (Newark), US-Central (Dallas), US-West (Fremont), US-Southeast (Atlanta), CA-Central (Toronto), EU-West (London), EU-Central (Frankfurt), AP-South (Singapore), AP-Northeast (Tokyo 2), AP-West (Mumbai), AP-Southeast (Sydney)
Time to detect: 1h24m
Time to resolve: 2h30m

Symptoms

From about 18:30 UTC on June 15, 2022, customers of the Linode Kubernetes Engine (LKE) who tried to upgrade a cluster or add nodes saw those operations fail silently, without an error. The status page opened an investigation of an LKE issue at 19:54 UTC, and the affected list covered LKE in eleven regions.

Symptom ? Upgrade stall

Root cause

Root cause ? Faulty deployment

Component ?Node pool

Contributing factor ?–

Trigger: An incremental update pushed to the LKE production environment at about 18:30 UTC.

Detection

The status page began investigating at 19:54 UTC, 1 hour 24 minutes after the update.

Remediation

Released a follow-up LKE update at about 21:00 UTC that contained the missing changes

Action items

Add steps and automation to the update process done

Lessons

–

Sources

https://status.linode.com/incidents/x14hy9s656ww · official-status

Related cases

IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regionsLinode · 2026
same root cause · shared Node pool
IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went downUK Ministry of Justice · 2025
same root cause · shared Node pool
IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accountsLinode · 2023
same root cause · shared Node pool
IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthyMicrosoft Azure · 2019
same root cause · shared Node pool