IL-0129 Hugging Face · 2025-10-30 (published)ai-infraaws
The training data was first stored on /fsx, a WekaFS network filesystem that keeps frequently read files in its local cache and evicts inactive files to S3 as capacity fills. The 24 TB SmolLM3 dataset, together with storage already used by several other teams, pushed Weka to its limit, and it began evicting dataset shards in the middle of training. Each evicted shard had to be fetched back, which stalled the run. The team found no way to pin the dataset folders as hot for the whole run, and streaming straight from S3 was slow, so the data was copied to the local NVMe RAID scratch storage on every node (8 x 3.5 TB drives per node).
Local storage brought its own problem: a replacement node had no data. Downloading the 24 TB from S3 with s5cmd took 3 hours; copying from a healthy node in the same datacenter with fpsync took 1.5 hours. The team then kept a spare node with the dataset preloaded in the Slurm reservation, so a failed node could be swapped at once; the spare ran evaluations or development jobs while idle.
The drops that remained came from the dataloader. SmolLM2 had been trained with steady throughput on a Megatron-LM-derived dataloader, TokenizedBytes, but SmolLM3 used nanotron's built-in dataloader, nanosets, which the team had never used for large-scale pretraining. nanosets built a single large index that grew with every training step, and at very large step counts the higher shared memory use caused the throughput drops. The same configuration run with TokenizedBytes in the team's internal SmolLM2 framework showed no drops on 48 nodes, so the team copied TokenizedBytes into nanotron instead of fixing the nanosets index, and throughput returned to target. TokenizedBytes read sequences in order within each document and made the loss noisier, which the team fixed by pre-shuffling the tokenized sequences offline for each epoch before relaunching.
Within hours of launching the full SmolLM3 pretraining run (3B parameters, 11T tokens, 384 H100 GPUs on 48 nodes), training throughput fell steeply, with repeated sharp drops. The team expected a steady 13,500 to 14,000 tokens per second, and none of the throughput ablations run beforehand on the same 48 nodes had shown drops.
Only three things differed from the ablations: the run now read the full training dataset of about 24 TB instead of subsets, the step count was set for 11T tokens instead of a 100B-token horizon, and the state of the hardware might have changed under sustained load. Hardware was the first suspect. Node-monitoring metrics showed that the large throughput fall lined up with spikes in disk read latency, which pointed at the data storage: the dataset sat on the shared WekaFS filesystem mounted at /fsx.
After the data was moved to each node's local NVMe RAID, the individual drops continued and became sharper, while hardware monitoring showed nothing unusual. The same drops appeared on a single node, whichever node was used, which ruled out hardware. Cutting the step count from about 3M to 32k made the drops smaller; longer step counts produced sharper and more frequent drops.
Symptom ? Throughput degradation
Root cause ? Capacity exhaustion
The team watched training throughput in tokens per second and saw it fall within hours of launch; node-monitoring metrics then tied the large fall to spikes in disk read latency.
–
Ablations on data subsets and a short step horizon did not exercise either cause: a shared cache-and-evict filesystem near its capacity, and a dataloader index that grows with the step count. Reproducing the drops on one node separated hardware from software, and varying only the step count pointed at the dataloader.
The team chose proven replacements over repairs under deadline pressure, taking an existing dataloader instead of fixing nanosets and offline shuffling instead of changing the dataloader, and warns that too many such shortcuts leave a patchwork system that is hard to maintain.
same root cause · shared Shared network filesystem