Service discovery

Cases, by severity

1. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip meshTarget · 2019 · sev1

   Root cause Dependency failure

2. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022

   Root cause Configuration change

3. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022

   Root cause Capacity exhaustion

4. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of trafficAirbnb · 2019

   Root cause Faulty deployment