IL-0042 UK Ministry of Justice · 2024-07-25kubernetesobservability
The Ministry of Justice Cloud Platform ships application logs from its live cluster with fluent-bit to Elasticsearch and OpenSearch, and users read them in Kibana. On 2024-07-25 neither store received logs for several hours, and the logs from that period were lost for good.
The ClusterIndexWritesBlocked alert for cp-live-app-logs fired on and off from 12:10, and Kibana stopped receiving logs entirely at 13:45. The team learned of it from a user at 14:27, started a call at 14:42 and declared the incident at 14:54. fluent-bit was failing with "could not enqueue into the ring buffer", and restarting it helped only for a few minutes. At 15:18 the team found OpenSearch out of disk space and raised it from 8000 to 12000; once that finished at 15:58, fluent-bit began processing logs again. As a temporary measure fluent-bit was also set to stop sending logs to OpenSearch while the root cause was investigated.
The report gives time to repair as 3h 8m and time to resolve as 4h 9m.
Kibana stopped showing new application logs at 13:45. At 14:27 a user told the Cloud Platform team in the #ask-cloud-platform channel that Kibana had received no logs since then.
Earlier signs had been there since 12:10, when the cp-live-app-logs alert ClusterIndexWritesBlocked first fired. Between 12:10 and 13:40 it started and recovered several times.
The first check, at 14:32, found no problems in the live monitoring namespace. At 14:55 the fluent-bit container logs showed "could not enqueue into the ring buffer". A rollout restart of all fluent-bit containers at 14:59 got some logs flowing again, but within a few minutes the same error returned. At 15:18 the team noticed that OpenSearch was out of disk space.
Symptom ? Monitoring outage
Root cause ? Capacity exhaustion
A user reported in #ask-cloud-platform at 14:27 that Kibana had received no logs since 13:45. The report records 12:10, when the ClusterIndexWritesBlocked alert first fired, as the first detection.
–
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent