IL-0174 Hyperconnect · 2023-05-02 (published)kubernetesobservability
Hyperconnect runs Fluent Bit as its log collector in Kubernetes, mostly as a DaemonSet and as a pod sidecar for applications that write especially many logs, feeding an EFK stack. After an upgrade to Fluent Bit 2.0, sidecars using the Kafka output plugin crashed repeatedly with a segmentation fault.
Core dump analysis showed that some event chunks arrived corrupted: after a valid record the buffer was filled with null bytes, and unpacking it produced objects that were not the expected array of timestamp and map. The Kafka output plugin passed whatever came back to produce_message(), which treated it as a map and walked memory with a bogus size until it crashed. The other output plugins already skipped records that were not maps; the Kafka plugin did not. The author could not find where the corruption came from, because event chunks are processed asynchronously and the core held no stack frame for that point; it was presumably introduced by the input_tail_file and filter_modify plugins in the configured pipeline.
The fix was to make the Kafka output plugin ignore non-map data before calling produce_message(). The problem did not reproduce reliably with the same input, so no test was written, but a custom build of the patched plugin running as the sidecar stopped the segmentation faults. The change was submitted upstream as fluent-bit pull request 6452 and merged eight days later.
Hyperconnect upgraded the Fluent Bit sidecar of one service with little user impact to Fluent Bit 2.0 and watched it for several days. Only the containers that used the Kafka output plugin were being restarted, each time on SIGSEGV. The stack trace printed at the crash pointed into the Kafka output plugin, but no core file was left behind, because Fluent Bit ran inside Kubernetes.
To capture one, the team mounted a hostPath volume into the pod and, on the node running the sidecar, set the kernel parameter core_pattern to "/tmp/cores/core.%e.%p.%h.%t". A test with a sleep process killed by SIGSEGV confirmed the dump appeared on both the pod and the host, and by the next morning a core named core.flb-out-kafka.0.1.some-pod-7dd6dc88c7-rz724.1669206388 had been written.
Opened with gdb in the fluent/fluent-bit:2.0.3-debug image (arm64), with the v2.0.3 source checked out, the backtrace showed the crash in produce_message() at plugins/out_kafka/kafka.c:189, "key = map->via.map.ptr[i].key;", called from cb_kafka_flush(). The variable map had type MSGPACK_OBJECT_FLOAT64 with the value 1669206388.2559929, a timestamp, but the code read it through the union as a map with size 1561354800, and the loop counter i had reached 165717.
One frame up, the unpacked result had type MSGPACK_OBJECT_POSITIVE_INTEGER, so flb_time_pop_from_msgpack had received input that did not match the event structure. The event chunk was 4911 bytes and the failure was at offset 859; from that offset on the buffer held only null bytes. Decoding the dumped chunk with Python msgpack showed one valid record followed by a long run of zeros.
Symptom ? Container crash loop
Root cause ? Software defect
The team monitored the upgraded sidecars for several days after the upgrade and saw that the containers using the Kafka output plugin were restarting with SIGSEGV.
–
A segmentation fault in a container can be debugged from its core file, but Kubernetes does not keep one by default: a volume mounted into the pod and a core_pattern pointing into it on the node are needed first. The debug image of the same version, run locally on the same CPU architecture with the matching source checkout, gives a gdb environment that matches production.
Core files contain part of the process memory and may hold sensitive data, so they should be shared with care.
same root cause · shared Telemetry agent
same root cause · shared Telemetry agent