IL-0139 Google Cloud · 2022-09-15kubernetesgcpnetworking
A race condition introduced in the Calico component affected GKE 1.22, 1.23 and 1.24 before 1.24.4-gke.800. On clusters with Calico network policy the race left pods stuck Terminating or Pending; when terminating pods, the Calico CNI plugin failed to tear down the pod network with an Unauthorized error on getting ClusterInformation. GKE worked with the Calico project on a fix.
The fix shipped first in GKE 1.24.4-gke.800 and was then made available in 1.23 and 1.22; clusters on the Rapid, Regular and Stable release channels running 1.22 or 1.23 were to upgrade automatically over the following weeks. Until then, the recommended action was to upgrade manually to 1.24.4-gke.800 or later or to ask Google Cloud Support for an internal patch; restarting the kubelet and calico-node got stuck pods moving, and in the first update disabling Calico network policy was offered for customers without a strong need for Calico. Google Cloud reported the issue resolved for all affected users as of 13:45 US/Pacific on 2022-09-29.
After upgrading to GKE 1.22 or later, clusters that use Calico network policy could have pods stuck in Terminating or Pending. Google Cloud posted the incident at 16:01 US/Pacific on 2022-09-15 as pods stuck terminating; from the 2022-09-21 update the description also named pods stuck Pending.
When terminating pods, the Calico CNI plugin reports: "Warning FailedKillPod 36m (x389 over 121m) kubelet error killing pod: failed to "KillPodSandbox" for "af9ab8f9-d6d6-4828-9b8c-a58441dd1f86" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod "myclient-pod-6474c76996" network: error getting ClusterInformation: connection is unauthorized: Unauthorized".
The affected versions were all 1.22 and all 1.23 GKE versions, and 1.24 versions before 1.24.4-gke.800. Only a small number of GKE clusters actually experienced stuck pods, and use of the cluster autoscaler increased the chance of hitting the race condition.
Symptom ? Pods stuck in TerminatingPods Pending
Root cause ? Software defect
–
A race in the Calico component can leave pods stuck Terminating, with FailedKillPod and KillPodSandbox errors that point at the network plugin, not at the workload. Restarting the kubelet and calico-node frees the stuck pods but does not remove the race; the fixed GKE version or Google's internal patch does. Clusters that use the cluster autoscaler are more likely to hit it.
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared Pod termination
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet