IL-0137 Lablup Inc · 2026-06-15 (published)ai-infra
The cluster has 63 NVIDIA HGX B200 nodes (504 GPUs; 60 nodes for training, 3 spare) and is run jointly by SK Telecom (cloud operations), Upstage (model development), Lablup (the Backend.AI infrastructure), NVIDIA (hardware) and VAST Data (storage). Training data and checkpoints live on a 2 PiB VAST Data E-Box mounted over NFS through a dedicated 200G RoCE storage network; the storage's cluster-wide maximum bandwidth is 700 GB/s for reads and 250 GB/s for writes.
When Upstage's Solar Open pre-training was deployed at 60 nodes, training initialisation from the storage took more than 8 hours. A joint diagnosis by the model-development, infrastructure, hardware and storage teams found that the application meant to do large sequential reads, but what actually reached the storage was fragmented small random I/O. With 60 nodes doing this at the same time, the combined access pattern saturated the storage's distributed metadata service, even though each node's own traffic looked normal.
The fix came from both sides. The application side sharded its Arrow files by rank, and the storage side replaced synchronous rm/unlink with asynchronous deletion and tuned readahead. Together these cut initialisation from more than 8 hours to less than 8 minutes. The report uses this case to argue that a storage and metadata bottleneck of this kind appears only at production scale and that root-cause analysis at this scale needs a metrics pipeline shared across organisations.
The 60-node NVIDIA B200 training configuration for the Solar Open pre-training run showed no major problem during initial validation on 2 to 4 nodes. Once the job ran at its full size of 60 nodes, training initialisation, which reads from the VAST Data storage and should have finished within minutes, took more than 8 hours, and storage throughput stayed far below the theoretical limit.
No single layer explained the delay. The application logs only showed that training initialisation was slow, and the storage metrics on their own did not show which access pattern was to blame. The storage NIC of each node received about 4 to 10 GiB/s, which looked unremarkable when each node was examined alone.
The cause became visible only when the application, scheduler, network and storage metrics of the five organisations running the cluster were lined up on one timeline. The Solar Open technical report, which describes the same problem, names I/O lock contention as a primary cause.
Symptom ? Throughput degradation
Root cause ? Capacity exhaustion
The problem showed up as training initialisation taking hours after the job was scaled to 60 nodes; application logs showed only the delayed initialisation.
–
Storage behaviour measured on 2 to 4 nodes did not predict behaviour on 60 nodes: a metadata bottleneck of this kind appears only at production scale, so small pre-tests cannot find it. Per-node network rates can look normal while the combined access pattern of all nodes overwhelms the storage metadata path. Finding the cause took metrics from the application, scheduler, network and storage on one shared timeline; monitoring by any single team was not enough. The report concludes that GPU capacity has to be provisioned together with a matching storage path.
same root cause · shared Shared network filesystem