IL-0002 Zalando · 2019-06-11kubernetesaws

Builder pods lost IAM credentials when kubelet fell behind the API rate limit

Kubelet talks to the API server under a request rate limit whose default is very low. With builder pods being created and deleted quickly on one node, kubelet could not update pod statuses fast enough. The credential broker kube2iam needs the pod IP that kubelet writes, so credential requests failed and builds died. Scaling the cluster up spread the pods and ended the incident. The permanent fix raised the kubelet QPS limit from the default 5 to 50 across all clusters.

Severity: –
Scope: internal continuous delivery platform
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

CI builds timed out and failed for close to an hour. Builder pods could not obtain AWS IAM credentials, and pod status in the API lagged behind what was running on the node.

Symptom ? High error rate Credentials unavailable Control plane lag

Root cause

Root cause ? Capacity exhaustion

Component ?kubelet Credential broker for pods Kubernetes API server

Contributing factor ?Unreviewed defaults Single point of failure

Trigger A rescale of the CI cluster left a single node for builder pods, and the rapid churn of pods on that node outran kubelet.

Detection

unknown

Remediation

scaled the cluster up so builder pods spread over more nodes

Action items

Raise the kubelet QPS limit from 5 to 50 in the global cluster configuration done

Lessons

The report states no lessons beyond the fix.

Sources

https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/postmortems/jun-2019-kubelet-qps.md · github · archived copy

References

Related cases

–