IL-0076 Fullstaq · 2019-06-01 (published)kubernetes
This is a meetup talk by Pieter Lange of Fullstaq, given as slides without notes. Dex ran with what the speaker calls bad defaults, an insane TTL, together with refreshing dashboards and an actual security bug. AuthRequest objects piled up in etcd until listing them timed out, and the speaker had seen hints of this earlier. The options he weighed were tuning etcd and API server timeouts, deleting the namespace and letting Kubernetes clean up, or deleting the objects directly in etcd.
He deleted the namespace. The reconciliation loops then listed all AuthRequests, timed out and started over, which drove master load to 60 and brought the masters down. He removed the kubernetes finalizer from the dex namespace by calling the API directly, with the kill-kube-ns tool, a step he marks NEVER DO THIS. That ended the outage, but about 0.5G of AuthRequests stayed in etcd, and he was fairly sure that creating a namespace named dex again would kill the cluster, so Dex was deployed to a new namespace with sane defaults. No business applications were affected.
Dex, the OIDC proxy that handled Kubernetes authentication, stored its data in the Kubernetes API (storage type kubernetes, inCluster: true) and had built up a huge number of AuthRequest objects. Listing all AuthRequests timed out, and without the list they could not be deleted.
After the dex namespace was deleted, the reconciliation loops kept listing all AuthRequests in order to delete them, timing out and repeating. Load on the masters rose to 60 and the masters went down. Monitoring could not be reached because authentication and port-forwarding no longer worked, while alerts kept arriving in Slack.
Root cause ? Capacity exhaustion
The speaker says he had got hints of the problem earlier. While the masters were down, plenty of alerts arrived in Slack, but monitoring could not be reached because authentication and port-forwarding no longer worked.
–
Dex with Kubernetes as its storage keeps every AuthRequest as an API object, and with a long TTL they pile up until the API cannot list them. Deleting a namespace that holds that many objects makes Kubernetes list them in a loop and can overload the masters. Monitoring that is reached through cluster authentication is lost when authentication is what broke.
same root cause · shared Kubernetes API server, etcd
same root cause · shared Kubernetes API server, etcd
same root cause · shared Kubernetes API server, etcd
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server
same root cause · shared Kubernetes API server