IL-0003 Monzo · 2017-10-27kubernetesaws
An earlier etcd reconfiguration had left the Kubernetes clients in the API server timing out, so the linkerd routing layer stopped receiving endpoint updates. The new deploy therefore never got traffic. Restarting every linkerd instance, done in good faith, made the whole platform lose routing at once, and a service object with no endpoints hit a parsing bug in the linkerd version in use. Recovery came from restarting the API servers, rolling linkerd forward to a fixed version and deleting the empty services. The outage lasted about an hour and a half.
Right after a deploy, every request to the payments service failed, and rolling back did not help. Within the hour, traffic stopped flowing between services across the platform and the app, balances and card payments were all affected.
Symptom ? High error rateStale endpoint routingService outage
Root cause ? Software defect
Trigger A deploy of the current-account payments service at 14:10, after an etcd cluster reconfiguration some weeks earlier.
The first failures were noticed within minutes of the deploy; most automated alerts fired only an hour later when the platform was already down.
Failures in a large distributed system are hard to diagnose, and well-meant manual action can widen them. The deployment and request-routing layers have to be trustworthy in a dynamic network.
–