IL-0051 thredUP · 2018-10-23 (published)kubernetesawsobservability

A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files

thredUP is an online consignment store for women's and children's clothes. Over about a year it moved a stack of roughly 30 to 40 services from EC2 to Kubernetes on AWS, run with kops, Terraform, Ansible, Jenkins, Helm and HAProxy. In a 2018 conference talk the CTO describes the problems found along the way; this case is the open files problem.

An infrastructure engineer worked with the vendor of a third-party log aggregation service whose log client sent the logs to the provider. Every container's log file needed a file descriptor in the client. When a new container started, the client opened a descriptor for the new log file and also opened new descriptors for the log files of the containers already running, leaving the old ones open. With three containers on a node, a fourth container left the first three containers' descriptors open twice. Because Kubernetes adds, removes and moves containers all the time, descriptors piled up, and on nodes with many containers each new one made the count jump.

Raising the settings the team found only delayed the failure. The vendor fixed the bug, and thredUP later moved away from that vendor. In the Q&A an infrastructure engineer confirmed the problem was in the application, not in Kubernetes tuning: the log client of the third-party provider was not ready for Kubernetes.

Severity

–

Scope

Kubernetes nodes running the log client; services on those nodes went down

Users affected

–

Region

–

Time to detect

–

Time to resolve

–