IL-0009 InfluxData · 2023-10-20 (published)kubernetes
Argo CD applied the generated file in order, so the later object with the duplicate name replaced the production Application and the old one's resources were pruned, including the stateful ones. The team stopped a reflex rollback that would have created an empty instance. They recreated the Kubernetes objects so the volumes reattached, restored etcd and Zookeeper from backups, rebuilt Kafka pod by pod while writes queued in its log, and let storage pods reindex from object storage. Full function returned about six hours after the merge with no data lost.
Every API endpoint answered not found. Writes, reads, scheduled tasks and data collection all stopped at once.
Root cause ? Configuration change
Trigger A merged pull request added objects to a generated configuration file in which an automation bug gave a new Argo CD Application the same name as the production one.
Monitoring did not catch it right away; the report does not say what did.
Monitoring has to notice this kind of change quickly, a written recovery plan beats a reflex rollback for stateful systems, and stateful resources need protection from pruning.
IL-0010 A dropped template delimiter made Argo CD delete every service worldwidesame root cause · shared Argo CD
IL-0010 — linked by the editor