IL-0142 Google Cloud · 2023-07-06kubernetesgcpobservability
For twelve days GKE clusters at version 1.24 and later wrote log messages that GKE should never have added. According to the preliminary analysis in Google's mini incident report, the cause was the rollout of a new version of the component responsible for processing logs. The extra messages counted against the customer's Logging API write quota, and some customers hit that quota, which can cause other log messages to be dropped.
While the fix was prepared, customers were told the messages could be ignored and offered two workarounds: request a higher Logging API quota, or add a log exclusion so the messages are not stored and billed. The exclusion matched resource.labels.container_name = "fluentbit-gke" or LOG_ID("fluentbit") together with the three message patterns. Google identified and tested a fix and rolled it out gradually to the affected clusters from 10 July; the incident ended at 07:27 US/Pacific on 18 July 2023.
From 07:13 US/Pacific on 6 July 2023, GKE customers saw unexpected additional error and warning entries in their cluster logs, written to the Cloud Logging project that holds the clusters. Standard and Autopilot clusters at GKE 1.24.x and later were affected.
The extra entries read "Failed to get record: decoder: failed to decode payload: EOF", "cannot parse '0609 05:31:59.491654' after %L" and "invalid time format %m%d %H:%M:%S.%L%z for '0609 05:31:59.490647'". Google's log exclusion query matched them on the fluentbit-gke container or the fluentbit log ID, and the last one also on the parser:glog plugin.
The entries consumed Logging API write quota and added to logging costs. For very large workloads, with thousands of nodes per project, they could exceed the quota, and other log messages could then be dropped; a subset of customers ran into quota exhaustion. The workloads running on the clusters were not affected.
Symptom ? Log flood
Root cause ? Faulty deployment
–
–
Log messages written by a managed logging agent count against the customer's own Logging API write quota and logging bill, so a faulty agent release can drop a customer's other logs. A log exclusion on the agent's container name and the message patterns keeps the noise out of storage until a fix arrives.
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent