IL-0071 Onkar Raghunath Shelke · 2026-09-17 (published)kubernetesobservability
The application image was built in one stage, so it carried the source, the build output and the whole node_modules tree including development dependencies: more than 40,000 files, over half of them in the @mui/icons-material package. containerd's content store deduplicates compressed layers by digest, but the overlayfs snapshotter unpacks every distinct layer into its own directory, so each new build with a different layer digest added another full copy of those files to every node that pulled it. The snapshots stay until the image is removed.
kubelet removes unused images only through image garbage collection, which starts when the image filesystem passes 85% of bytes and stops at 80%. It never looks at file counts. kubelet does have inode thresholds on Linux, nodefs.inodesFree and imagefs.inodesFree at 5% free, but those are hard eviction thresholds: the first action kubelet takes when inodes run low is to evict pods. Because the files were small, inodes were being used up much faster than bytes. The author works out from the df figures that the root filesystem holds about 16.3 million inodes, more than the stock mkfs ratio of one inode per 16,384 bytes allows for its size, and that it was probably formatted at about 8,192 bytes per inode; this is arithmetic, not a capture, because the author no longer has access to the cluster. Kubelet, containerd and the node image all ran on defaults.
The alert caught the trend at 67% inode use, before kubelet's eviction threshold was reached. The author's fixes are a multi-stage Dockerfile that ships only the compiled output, a flat inode usage alert alongside the predictive one, and pruning unused images on the node.
A Kubernetes worker node paged with NodeFilesystemFilesFillingUp, the node_exporter mixin alert that predicts a filesystem will run out of inodes. The engineer checked disk space first: df -h showed the root filesystem 83% used, 103G used and 22G available. df -i showed inodes 67% used, 11M used and 5.3M free.
The disk was fuller than the inode table, yet the inode table was what paged. The alert is built on predict_linear, so it fires on how fast inodes are being consumed, not on the current level; 67% was not a problem in itself, the rate of growth was.
du --inodes -xS /var showed where the files were: not in logs or pod volumes but in containerd's snapshot store, /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/. Several snapshot directories each held the same path, app/node_modules/@mui/icons-material, with 21,553 files apiece.
Symptom ? Node disk full
Root cause ? Capacity exhaustion
The NodeFilesystemFilesFillingUp alert from the node_exporter mixin fired on the predicted inode trend while inode use was at 67%.
–
kubelet's background cleanup works on bytes only. A workload heavy in small files gets no early intervention for inodes, and the first thing kubelet does at 5% free inodes is evict pods. Watch inode use on nodes separately from disk use, and keep dependency trees out of runtime images.
same root cause · shared Node root disk, Container runtime, kubelet
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet
same root cause · shared kubelet