Jobs and CronJobs

Cases, by severity

1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025

   Root cause Operator error

2. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018

   Root cause Configuration change

3. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was downUniverse · 2017

   Root cause Capacity exhaustion