From the VAST Data edition of October 31, 2026
LLNL study on the Tuolumne production system finds VAST avoids all-reduce contention but still shows heavy-tailed DataLoader stalls in 12% of iterations
Lawrence Livermore National Laboratory (LLNL) researchers (paper accepted at PDSW'26) · National-laboratory research computing (transferable to university and state AI clusters) · United States (Livermore, California)
- Publisher
- arXiv (2609.06506v2)
- Original publication
- 2026-09-19
- Source retrieved
- 2026-11-01
What happened
On LLNL's Tuolumne (HPE Slingshot-11, four nodes with 16 AMD MI300A GPUs, a real GNN training workload), DataLoader waits were 15 ms median on VAST but spiked to multiple seconds in 12% of VAST iterations versus 28% for Lustre. VAST, mounted over NFSv3/TCP, sits in a separate Slingshot traffic class, which largely spared all-reduce from contention, but its data path still crosses the shared fabric. Node-local NVMe staging (DYAD) was 1.06x faster than VAST over one epoch and 1.31x once the cache warmed, and 7.4x faster than Lustre.
Why it matters
Relevant to university, state-funded and national-lab-adjacent AI clusters that pair VAST with a parallel file system, including the Empire AI Gamma RFI issued October 29, 2026, which asks how storage will interoperate with Beta's VAST tier. Why now: the paper was revised in September and is accepted at PDSW'26 (held with SC26 in November); it offers independent measured evidence on exactly the storage-fabric questions that state AI consortia are now putting into procurements.
Evidence and measured results
Independent measured study by national-lab researchers on a production system, comparing Lustre, VAST and node-local staging with the same workload. Sample is narrow: one GNN workload, four nodes, one fabric and VAST over NFSv3/TCP. DYAD is the authors' own system, so the comparison is not neutral toward it. The arXiv version is a preprint of a workshop paper.
Limitations and uncertainty
Single workload and small scale; results depend on Slingshot traffic classes and NFSv3/TCP mounts and may differ with RDMA or other fabrics. Authors evaluate their own mitigation. Not a SLED deployment; transferability to university or state clusters is an interpretation. Original submission September 6, 2026; v2 September 19.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-31; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- AI research clusters waste GPU time when training waits on shared storage traffic.
- Stakeholders
- Research computing director
- HPC storage and network engineers
- Lead AI research groups
- Procurement team drafting storage requirements
- Discovery questions
- Does storage traffic share the fabric and traffic class with GPU collectives?
- Do you track DataLoader stalls or only aggregate bandwidth?
- Would node-local staging fit your data-handling rules?
- Value hypothesis
- Placing VAST in a separate traffic class and adding caching may reduce training stalls and communication interference; evidence is one workload at four nodes.
- Potential engagement
- A scoped storage-path assessment measuring stall and tail-latency behaviour on the customer's own training workload.
- Unsupported claims
- That VAST eliminates I/O stalls
- Speedups at large scale or other workloads
- General superiority over Lustre beyond this configuration
Pre-sales engineering
Role takeaway
- Fit
- Fits GPU clusters on Slingshot or similar shared fabrics using VAST over NFS alongside a parallel file system.
- Architecture and integration
- VAST mounted over NFSv3/TCP on the Ethernet face (TC0) of Slingshot-11 NICs; Lustre and RCCL on the kfabric face (TC1+); optional node-local NVMe staging via DYAD.
- Prerequisites
- Knowledge of fabric traffic-class configuration
- Representative training workload
- Node-local NVMe capacity if staging is used
- Constraints
- One GNN workload on four nodes
- Results specific to Slingshot traffic classes and NFSv3/TCP
- Authors evaluate their own mitigation
- Security
- Study does not address security; staging data on compute nodes requires review for sensitive datasets.
- Proof of value
- Measure DataLoader wait distribution on VAST under production concurrency
- Test all-reduce latency with and without concurrent storage I/O
- Compare stall rates with and without node-local staging
Delivery
Role takeaway
- Work
- Operationalize monitoring and configuration of storage paths for AI training on shared fabrics.
- Dependencies
- Fabric administrators
- Storage team
- Research users willing to share workload profiles
- Ownership
- Research computing operations, with storage and network engineering jointly responsible for the storage path.
- Skills and adoption
- Staff need to interpret iteration-time tails and fabric traffic classes, beyond storage capacity management.
- Governance checkpoints
- Acceptance testing including tail-latency metrics
- Change control for fabric and mount settings
- Data-handling review for node-local staging
- Acceptance criteria
- Proposed: share of iterations with DataLoader wait over 1 s below an agreed threshold
- Proposed: all-reduce latency under concurrent I/O within agreed factor of baseline
- Proposed: retest after fabric or client upgrades
- Risks
- Bandwidth-only acceptance hiding stall problems
- Fabric changes reintroducing contention
- Transferring small-scale results to large deployments
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Traffic-class separation for VAST NFS over TCP avoids collective-communication interference but does not remove tail latency, because storage I/O still crosses the shared fabric. Node-local staging or caching in front of VAST can reduce stalls for repeated-epoch training.
Governance
Who approves, reviews and stays accountable for outcomes?
Procurements should require tail-latency and stall metrics under representative training, not only aggregate bandwidth ratings, and should state fabric and traffic-class placement for storage.
Security and privacy
What data, permissions and controls need testing?
Not addressed by the study; node-local staging copies data onto compute nodes, which matters for regulated datasets and should be considered in data-handling plans (interpretation).
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Not addressed; diagnosing tail-latency issues requires specialised HPC staff skills.
Procurement
What should contracts, pricing and exit terms secure?
Supports requiring tail-latency and contention metrics on representative workloads in storage RFPs rather than rated bandwidth alone.
Operating model
Which teams own the service once it runs?
Storage and network teams must jointly own the training data path; staging layers add operational components.
Publication history
- 2026-10-31VAST Data · Issue 032 resources
Stable resource ID: llnl-tuolumne-shared-fabric-vast-dataloader-stalls-pdsw26