IL-0114 CoW Protocol · 2024-04-19kubernetesawsnetworking
A change to the liquidity fetching logic of CoW Protocol's baseline liquidity caused a performance regression. Higher memory and CPU use got core services evicted frequently, so the team expanded the capacity of the production Kubernetes cluster. The Infrastructure as Code (IaC) run for that expansion also updated the VPC CNI plugin, the AWS component that gives each pod an IP address from the VPC. The CNI version was not hardcoded in the IaC, package versions were not properly pinned, and the IaC library version had recently changed, so the plugin was updated automatically and the result was a CNI version mismatch.
With the mismatched CNI, newly deployed pods never received an IP address and stayed Pending, and updates and pod terminations no longer worked. Settlements stopped at 16:40 UTC. The team identified the CNI version mismatch as the cause at 17:03 UTC, started a cluster upgrade around 19:00 UTC, and set up new node groups with a new version of the networking plugin. Services were restored at 20:08 UTC, about 3.5 hours after the outage began.
At 16:40 UTC on 19 April 2024 CoW Protocol's mainnet production environment stopped producing settlements; the outage officially began then.
Core services had already been evicted often, because a change to the liquidity fetching logic of the baseline liquidity had raised memory and CPU use. After the networking plugin was updated, every restarted pod stayed in Pending because it could not get an IP address, so each restart left the system less available than before. Updates and pod terminations stopped working as well.
Symptom ? Pods PendingIP address exhaustionPod evictionService outage
Root cause ? Configuration change
The CNI version mismatch was identified as the cause at 17:03 UTC, 23 minutes after settlements stopped.
–
Versions in IaC scripts should be specified explicitly so an automation run cannot update a component such as the CNI plugin unintentionally. Changes in production need change management that is easy to revert.
same root cause · shared AWS VPC CNI, Node pool
same root cause · shared AWS VPC CNI, Node pool
same root cause · shared Node pool
same root cause · shared Node pool
same root cause · shared AWS VPC CNI
same root cause · shared Node pool