IL-0066 Pusher · 2019-05-21 (published)kubernetes
In April 2018 an engineer at Pusher deployed a change to the bridge's ConfigMap without changing the deployment that used it. The configuration was valid, passed linting, and the bridge would have started with it, but it broke routing to the backend services. Updating a ConfigMap in Kubernetes does not roll the pods of a deployment that references it: the files mounted in running pods are updated in place, and an application that does not reload its configuration, such as the bridge, keeps running with the old one. Nothing happened until three or four weeks later, when a cluster upgrade replaced the bridge pods and each new pod read the broken configuration. When the last old pod was gone, the whole platform was down.
The speaker says this kind of incident happened several times during 2018. To stop it, Pusher built Wave, an open-source controller that watches the ConfigMaps and Secrets referenced by deployments, stateful sets and daemon sets, and writes a hash of their data into the pod template. Any configuration change then triggers a normal rollout, so a bad configuration fails immediately instead of weeks later.
During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.
All customer traffic passes through a load balancer, then nginx, then the bridge, an in-house layer 7 proxy that handles Pusher-specific authentication, HTTP/2 and WebSocket upgrades, before reaching the product services. As the upgrade replaced nodes, bridge pods were recreated one after another, and once every bridge pod was running the new configuration, routing to the backend services stopped working.
Symptom ? Service outageHigh error rate
Root cause ? Configuration change
–
–
A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.
–