IL-0055 Target · 2018-09-20 (published)kubernetes
The speaker began her career at Target, which started running Kubernetes about three years before the talk for thousands of application developers who were eager to use it but, in her words, not ready for it. She goes through the mistakes she saw there, several of which took a cluster down, and calls the failing CronJob the most common cause of a cluster outage she has seen.
A CronJob's default concurrency policy lets a new run start while earlier runs still exist. When the job fails, for example because the database it writes to is not there, each run keeps restarting and the next scheduled run adds another restarting pod instead of replacing the old one. A job scheduled every minute that fails for three days produces 4,320 restarting pods. On this cluster the pods reached about four thousand, the autoscaled cluster grew to a few hundred virtual CPUs, and the cluster went down.
Her advice is to set the CronJob concurrency policy to Replace or Forbid, and to make applications fail well: use init containers, readiness and liveness probes, and retry a missing dependency such as a database with exponential back-off for a bounded time instead of crashing and restarting against it in a tight loop.
A Kubernetes cluster went down after one CronJob kept failing and its pods piled up. Because each failed run kept restarting instead of being replaced by the next one, the pods accumulated into the thousands. The speaker's arithmetic for a job scheduled every minute that fails for three days is 4,320 restarting pods.
The cluster was autoscaled, so it grew with the pods. By the time the pods reached about four thousand, the cluster had a few hundred virtual CPUs attached, which also cost a lot of money.
Symptom ? Service outage
Root cause ? Configuration change
–
–
–
When deploying a CronJob, set its concurrency policy to Replace or Forbid; with the default, failing runs accumulate and restart together.
An application should handle a missing dependency with probes and bounded back-off rather than crash in a loop, which also keeps it from hammering a dependency that is already struggling.
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared Node pool