{"resourceId":"llnl-tuolumne-shared-fabric-vast-dataloader-stalls-pdsw26","versions":[{"version":"external-6e57f229ff422b9d67878fe82bfc0fe0f93f8fc0a6265312bd7b865c1eb70111","resource":{"id":"llnl-tuolumne-shared-fabric-vast-dataloader-stalls-pdsw26","title":"LLNL study on the Tuolumne production system finds VAST avoids all-reduce contention but still shows heavy-tailed DataLoader stalls in 12% of iterations","organization":"Lawrence Livermore National Laboratory (LLNL) researchers (paper accepted at PDSW'26)","sector":"National-laboratory research computing (transferable to university and state AI clusters)","geography":"United States (Livermore, California)","publishedAt":"2026-09-19","publicationDate":"2026-09-19","eventDate":null,"sourceName":"arXiv (2609.06506v2)","sourceLabel":"Wang, Wu, Kim and Yeom, 'Sharing a Fabric with Collective Communication' (accepted at PDSW'26)","sourceUrl":"https://arxiv.org/abs/2609.06506","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["infrastructure","operating-model"],"finding":"On LLNL's Tuolumne (HPE Slingshot-11, four nodes with 16 AMD MI300A GPUs, a real GNN training workload), DataLoader waits were 15 ms median on VAST but spiked to multiple seconds in 12% of VAST iterations versus 28% for Lustre. VAST, mounted over NFSv3/TCP, sits in a separate Slingshot traffic class, which largely spared all-reduce from contention, but its data path still crosses the shared fabric. Node-local NVMe staging (DYAD) was 1.06x faster than VAST over one epoch and 1.31x once the cache warmed, and 7.4x faster than Lustre.","sledRelevance":"Relevant to university, state-funded and national-lab-adjacent AI clusters that pair VAST with a parallel file system, including the Empire AI Gamma RFI issued October 29, 2026, which asks how storage will interoperate with Beta's VAST tier. Why now: the paper was revised in September and is accepted at PDSW'26 (held with SC26 in November); it offers independent measured evidence on exactly the storage-fabric questions that state AI consortia are now putting into procurements.","evidence":"Independent measured study by national-lab researchers on a production system, comparing Lustre, VAST and node-local staging with the same workload. Sample is narrow: one GNN workload, four nodes, one fabric and VAST over NFSv3/TCP. DYAD is the authors' own system, so the comparison is not neutral toward it. The arXiv version is a preprint of a workshop paper.","architectureImplications":"Traffic-class separation for VAST NFS over TCP avoids collective-communication interference but does not remove tail latency, because storage I/O still crosses the shared fabric. Node-local staging or caching in front of VAST can reduce stalls for repeated-epoch training.","governanceImplications":"Procurements should require tail-latency and stall metrics under representative training, not only aggregate bandwidth ratings, and should state fabric and traffic-class placement for storage.","securityPrivacyImplications":"Not addressed by the study; node-local staging copies data onto compute nodes, which matters for regulated datasets and should be considered in data-handling plans (interpretation).","caveats":"Single workload and small scale; results depend on Slingshot traffic classes and NFSv3/TCP mounts and may differ with RDMA or other fabrics. Authors evaluate their own mitigation. Not a SLED deployment; transferability to university or state clusters is an interpretation. Original submission September 6, 2026; v2 September 19.","streamIds":["vast-data"],"roles":{"sales":"This LLNL study gives an independent, measured view of VAST in AI training: fewer stalls than Lustre and much less interference with GPU communication, but not stall-free. Use it with university and state AI consortia to set honest expectations and to position storage-plus-caching designs rather than platform claims. Do not present it as a VAST benchmark win; it is one workload at small scale, and the authors' own staging layer beat VAST once caches warmed.","engineering":"The engineering lesson is fabric placement. On Tuolumne, VAST over NFSv3/TCP in a separate Slingshot traffic class largely protected all-reduce, yet 12% of iterations still had multi-second DataLoader waits because storage traffic crossed the shared fabric. Designs should specify traffic classes or separate networks for storage, consider node-local staging for repeated-epoch training, and measure tail latency, not just bandwidth, with representative workloads before acceptance.","delivery":"Delivery teams should build stall monitoring into acceptance and operations: DataLoader wait distributions, iteration-time tails and all-reduce interference under realistic concurrency. Where repeated epochs dominate, offering a node-local staging or caching option can cut stalls, but it adds data-handling duties on compute nodes. Document fabric and traffic-class settings for storage mounts so changes do not silently reintroduce contention, and retest after fabric or client upgrades."},"roleDetails":{"sales":{"customerProblem":"AI research clusters waste GPU time when training waits on shared storage traffic.","stakeholders":["Research computing director","HPC storage and network engineers","Lead AI research groups","Procurement team drafting storage requirements"],"discoveryQuestions":["Does storage traffic share the fabric and traffic class with GPU collectives?","Do you track DataLoader stalls or only aggregate bandwidth?","Would node-local staging fit your data-handling rules?"],"valueHypothesis":"Placing VAST in a separate traffic class and adding caching may reduce training stalls and communication interference; evidence is one workload at four nodes.","potentialEngagement":"A scoped storage-path assessment measuring stall and tail-latency behaviour on the customer's own training workload.","unsupportedClaims":["That VAST eliminates I/O stalls","Speedups at large scale or other workloads","General superiority over Lustre beyond this configuration"]},"engineering":{"fit":"Fits GPU clusters on Slingshot or similar shared fabrics using VAST over NFS alongside a parallel file system.","architectureIntegration":"VAST mounted over NFSv3/TCP on the Ethernet face (TC0) of Slingshot-11 NICs; Lustre and RCCL on the kfabric face (TC1+); optional node-local NVMe staging via DYAD.","prerequisites":["Knowledge of fabric traffic-class configuration","Representative training workload","Node-local NVMe capacity if staging is used"],"constraints":["One GNN workload on four nodes","Results specific to Slingshot traffic classes and NFSv3/TCP","Authors evaluate their own mitigation"],"security":"Study does not address security; staging data on compute nodes requires review for sensitive datasets.","proofOfValue":["Measure DataLoader wait distribution on VAST under production concurrency","Test all-reduce latency with and without concurrent storage I/O","Compare stall rates with and without node-local staging"]},"delivery":{"work":"Operationalize monitoring and configuration of storage paths for AI training on shared fabrics.","dependencies":["Fabric administrators","Storage team","Research users willing to share workload profiles"],"ownership":"Research computing operations, with storage and network engineering jointly responsible for the storage path.","skillsAdoption":"Staff need to interpret iteration-time tails and fabric traffic classes, beyond storage capacity management.","governanceCheckpoints":["Acceptance testing including tail-latency metrics","Change control for fabric and mount settings","Data-handling review for node-local staging"],"acceptanceCriteria":["Proposed: share of iterations with DataLoader wait over 1 s below an agreed threshold","Proposed: all-reduce latency under concurrent I/O within agreed factor of baseline","Proposed: retest after fabric or client upgrades"],"risks":["Bandwidth-only acceptance hiding stall problems","Fabric changes reintroducing contention","Transferring small-scale results to large deployments"]}},"retrievedAt":"2026-10-31T22:04:00-05:00","enrichedAt":"2026-10-31T22:07:02-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Not addressed; diagnosing tail-latency issues requires specialised HPC staff skills.","procurementImplications":"Supports requiring tail-latency and contention metrics on representative workloads in storage RFPs rather than rated bandwidth alone.","operatingModelImplications":"Storage and network teams must jointly own the training data path; staging layers add operational components.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.06506","referenceExcerpt":"spikes to multiple seconds in 28% of Lustre iterations and 12% of VAST iterations","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}