IL-0031 Zalando · 2018-12-05 (published)kubernetesaws
This is the first of six incidents in Henning Jacobs' talk on running about 100 Kubernetes clusters at Zalando. A team's CronJob manifest, which the talk calls innocent, put concurrencyPolicy: Forbid, successfulJobsHistoryLimit and failedJobsHistoryLimit under the pod template's spec, where they do not belong. The fixed manifest moves those three fields to the CronJob spec, adds activeDeadlineSeconds: 120 to the job spec, and changes the schedule to "7 8-18 * * Mon-Fri". The talk shows this manifest after a slide on the number of pods in the cluster and the API server being OOM-killed.
Because Skipper takes its routes from the API server, the API server outage became an ingress outage: customer traffic failed and the IAM service returned 404. The lessons note that the AWS ALB in front routes traffic to all hosts when all of them report unhealthy. The talk's lessons are to keep the ingress healthy and serving its last known routes during API server problems, and to cap the number of pods with a ResourceQuota.
Customers were affected, and Zalando's IAM service answered requests with HTTP 404. The talk shows this on slides titled "Customer impact" and "IAM returning 404", followed by a slide on the number of pods in the cluster.
The Kubernetes API server had been OOM-killed. Traffic reached applications through an AWS ALB and then Skipper, the HTTP router that Zalando runs as its ingress controller on the nodes, and Skipper loads its routes from the API server. With the API server gone, Skipper lost its source of routes.
The lessons slide adds that when every host behind an AWS ALB reports unhealthy, the ALB sends traffic to all hosts anyway, and the ingress did not stay healthy or keep its last known routes while the API server was down.
Symptom ? High error rateControl plane unresponsiveOOMKilled
Root cause ? Capacity exhaustion
–
An AWS ALB routes traffic to all hosts when all hosts report unhealthy, so the ingress should stay healthy during API server problems. An ingress controller that reads routes from the API server should keep its last known set of routes. Use a quota for the number of pods.
same root cause · shared Ingress controller
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Ingress controller
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server