IL-0043 UK Ministry of Justice · 2023-08-04kubernetesobservability
On the Ministry of Justice Cloud Platform, Fluent Bit ships application logs to a logging cluster that users read through Kibana. On the morning of 4 August 2023 users reported long periods of missing logs. The team traced it to a change in how the newer Fluent Bit version drops chunks. The platform lost application logs for a period of time.
Restarts, a newer Fluent Bit version and a bigger memory buffer were tried first. The fix was to handle buffer overflow by letting Fluent Bit write to the file system, merged and deployed to the live cluster on 9 August, and to flush logs in smaller chunks, added on 10 August. The incident was repaired at 12:28 and resolved at 14:47 on 10 August; the report gives the time to repair as 33h 14m and the time to resolve as 35h 33m.
At 09:14 on 4 August 2023, users of the Cloud Platform reported in the #ask-cloud-platform Slack channel that Kibana was missing their application logs for long periods.
The platform team began investigating at 10:03 and restarted the Fluent Bit pods; the incident was declared at 10:09. At 12:03 they found that the newer Fluent Bit version had changed its strategy for dropping chunks. The team moved to another Fluent Bit version at 16:00 the same day and enlarged the Fluent Bit memory buffer on 7 August; the repair came later, with changes to how Fluent Bit buffers and flushes logs.
Symptom ? Monitoring outage
Root cause ? Capacity exhaustion
Users reported long periods of missing logs in Kibana in the #ask-cloud-platform Slack channel at 09:14; the incident was declared at 10:09.
–
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent