Strategic PartnersIssue 585 records
NVIDIA
A cluster-operations edition. On Nov 2 NVIDIA open-sourced two DSX OS tools for distributed GPU training: Recovery, which detects, isolates and restarts failed jobs (Slurm first, monitor mode by default), and Attribution, a rules-based root-cause tool. It also published Preview Kubernetes recipes for Vera Rubin NVL72. NVIDIA reports activity metrics but no accuracy or goodput results. An October FailBench preprint finds no fault-tolerance mechanism wins everywhere. April reporting on NVIDIA's control of Slurm adds a vendor-dependency question for mixed-vendor university and state HPC sites. No new U.S. state, local or education NVIDIA deployment was found for Nov 2.
- Evidence records
- 5
- Cross-source patterns
- 2
- Evidence classes
- 3 vendor claim1 academic research1 independent reporting
- Outcomes
- 3 emerging1 mixed1 cautionary
- Source freshness
- 3 new this fortnight1 recent1 older, newly relevant
- Research completed
- 2026-11-03
Choose a role to see its takeaway beside every record in the ledger.
Patterns across the evidence
Synthesis · Lighthouse Advisory interpretation
NVIDIA's new open-source recovery and diagnosis tools report activity, not outcomes, so sites must measure the benefit themselves
On Nov 2 NVIDIA open-sourced Recovery and Attribution for distributed training clusters. Recovery's reported numbers are detection counts and action latencies from one cluster, and NVIDIA notes action latency is not the time until training resumes. Attribution gives no accuracy figures and says its quality depends on telemetry completeness. The independent FailBench study finds no fault-tolerance mechanism is best across architectures and that higher throughput after failures did not mean better training progress. Together they point to monitor-mode pilots scored on training progress and diagnosis accuracy.
Operating questionBefore letting software restart or kill research jobs, what accuracy and training-progress measures will a research-computing centre require from a monitor-mode pilot?
Supporting evidenceNVIDIA open-sources Recovery, a cluster-level service that detects, isolates and restarts failed training jobs, starting with SlurmNVIDIA open-sources Attribution, its rules-based tool for finding the root cause of distributed AI job failures, and says results depend on telemetry qualityKhaled Aljbab and Amine Barrak (arXiv preprint)
NVIDIA's operations tooling lands first on Slurm, the scheduler it now controls, sharpening the vendor-dependency question for mixed-vendor sites
NVIDIA Recovery supports Slurm only for now, and the Attribution Blueprint ships Slurm and Kueue connectors. NVIDIA has owned Slurm developer SchedMD since December 2025, and analysts quoted by InfoWorld say this gives NVIDIA soft power over a scheduler used on about 60% of supercomputers, while NVIDIA pledges neutrality and the GPL allows forking. No favouritism has been shown. The point for public HPC operators is that adopting NVIDIA's tooling on Slurm adds to an NVIDIA-steered stack.
Operating questionFor each NVIDIA-led component a university or state HPC centre adopts on Slurm, what is the exit path and how will it check support parity for non-NVIDIA hardware?
Supporting evidenceNVIDIA open-sources Recovery, a cluster-level service that detects, isolates and restarts failed training jobs, starting with SlurmNVIDIA open-sources Attribution, its rules-based tool for finding the root cause of distributed AI job failures, and says results depend on telemetry qualityInfoWorld; NVIDIA; SchedMD
Evidence ledger
Full record · every source keeps its link and limitations
NVIDIA open-sources Recovery, a cluster-level service that detects, isolates and restarts failed training jobs, starting with Slurm
NVIDIA released NVIDIA Recovery, part of its DSX OS software, under Apache 2.0. It runs a four-stage loop (detect, attribute, isolate, restart or stop) across a cluster's training jobs, using application logs, GPU metrics and scheduler signals. It supports Slurm today through a Helm-deployed Advisor Controller (in-cluster or SSH mode); Kubernetes support is on the roadmap. New deployments start in monitor mode, recording what they would have done without acting, and every action goes to a persistent audit trail. NVIDIA reports that on one training cluster from July 9 to October 9 it covered nearly 200 jobs per day, logged about 8,550 anomaly detections, and had mean detection and recovery-action latencies of 69 and 94 seconds. NVIDIA says IBM Research ran it in monitor mode before enabling autonomous action, and neocloud Verda contributed a VictoriaLogs connector.
Why it matters, evidence and limitations
- Why it matters
- University research computing centres and state HPC facilities that run Slurm for shared GPU clusters lose GPU-hours when multi-node jobs fail and staff restart them by hand. This is a free, open-source option they can trial in observe-only mode. Relevance is limited to sites running large distributed training, not most agency AI use.
- Evidence and measured results
- Vendor blog with one production observation window on a single NVIDIA cluster (detections and latencies only). No figures on false positives, recovery success rate or goodput gained; the IBM Research and Verda mentions are vendor-reported.
- Limitations and uncertainty
- NVIDIA's own metrics; recovery-action latency ends at the action, not when training resumes. Slurm only for now. No independent evaluation.
NVIDIA open-sources Attribution, its rules-based tool for finding the root cause of distributed AI job failures, and says results depend on telemetry quality
NVIDIA open-sourced NVIDIA Attribution, which it says it has used in production for years across hundreds of thousands of GPUs, analysing millions of workloads each week. It correlates workload-scoped application logs, syslogs, GPU driver and Fabric Manager messages, scheduler history and metrics, and uses deterministic rules to separate the root cause from downstream symptoms. For example, it identifies an XID 149 NVLink signal-integrity fault on one node as the cause of NCCL errors on other ranks. It ships as a self-hosted REST service plus a Python reference Blueprint with connectors for Slurm, Kueue, Loki, local files, Kubernetes pod logs and PromQL, and output to stdout or SQLite. When several causes fit the evidence, the result is marked ambiguous, and NVIDIA says attribution quality depends on telemetry completeness.
Why it matters, evidence and limitations
- Why it matters
- Useful to university and state HPC teams that spend staff time working out whether a failed GPU job was hardware, system software or user code. The deterministic, inspectable rules suit public operators who must explain incidents. Limited relevance outside sites running multi-node GPU work.
- Evidence and measured results
- Vendor blog with a synthetic worked example and an unquantified production anecdote (fabric port patterns in one RDMA cluster). No accuracy, precision or time-to-diagnosis figures.
- Limitations and uncertainty
- No published accuracy metrics; rules are tuned to NVIDIA hardware failure modes (e.g., GB200 priority rules); the worked example uses synthetic data.
NVIDIA's AICR adds Preview Kubernetes recipes for Vera Rubin NVL72, validated on early hardware, and lists the driver, NIC and topology changes from GB200
NVIDIA's AI Cluster Runtime (AICR) adds a Vera Rubin NVL72 recipe family (RKE2, Ubuntu, training, inference and Dynamo variants), validated end to end on early Vera Rubin clusters with GPU Operator v26.7.0, DRA driver v0.5.0 and a host-installed R615.23 driver. The post lists changes from GB200 NVL72: a new driver branch (R615.23 tested, not a confirmed minimum), ConnectX-9 SuperNICs and BlueField-4 DPUs replacing ConnectX-7 and BlueField-3, a different GPU-to-NIC topology, new MIG profiles not covered by the validation, and Kubernetes 1.34.1 or later. Early validation was blocked by a hardware/fabric issue that kept one tray out of the NVLink fabric until reseated. The recipes ship as Preview, without the production support and lifecycle qualification of Supported recipes.
Why it matters, evidence and limitations
- Why it matters
- Relevant to state, university and national-lab-adjacent buyers planning next-generation NVIDIA systems: it shows Vera Rubin will require new drivers, NICs, DPUs and Kubernetes versions, and that the software path is still pre-release. Useful for procurement timing, not for current deployments.
- Evidence and measured results
- Vendor validation on early-access reference clusters; one bus-bandwidth gate reported. Preview status stated by NVIDIA.
- Limitations and uncertainty
- Pre-release platform and Preview recipes; MIG not validated; tested versions are not stated minimums.
FailBench study finds no fault-tolerance mechanism is best across distributed training architectures; costs range from 3.7% to 176%
FailBench evaluates 142 combinations of seven distributed training architectures, eight crash-fault-tolerance mechanisms plus a no-fault-tolerance baseline, and single, concurrent and cascading fail-stop failures on an 8xV100 cluster. No mechanism is best everywhere: on one architecture, disk checkpointing has the lowest steady-state overhead (0.5%), in-memory replication restores fastest (~17 ms), and just-in-time checkpointing avoids periodic cost but takes ~0.9 s on failure. In-memory replication overhead ranges from 3.7% to 176% depending on architecture. Gossip training raised sample throughput 17.9% after losing a worker but showed no detectable improvement in loss progress. The authors release a decision framework and the harness.
Why it matters, evidence and limitations
- Why it matters
- Independent counterweight to vendor recovery tools released Nov 2: university HPC centres should expect the best recovery approach to depend on their workloads, and should measure training progress, not throughput or action latency alone. Published Oct 6; newly relevant because of NVIDIA's Recovery release.
- Evidence and measured results
- Preprint, not peer reviewed; small 8xV100 testbed; fail-stop failures only.
- Limitations and uncertainty
- Small, older-GPU testbed; results may not transfer to large NVLink clusters; does not evaluate NVIDIA Recovery.
Analysts question whether NVIDIA's control of Slurm developer SchedMD could tilt the open-source scheduler toward NVIDIA hardware
InfoWorld, building on Reuters reporting from five anonymous sources, reports industry concern that NVIDIA, which acquired SchedMD in December 2025, could favour its own hardware in Slurm through roadmap and code-review control. Slurm runs on roughly 60% of the world's supercomputers, including government systems for weather and national security research. NVIDIA committed to keep Slurm open-source and vendor-neutral. A TechInsights analyst said SchedMD's control gives NVIDIA soft power rather than hard lock-in, and that CUDA support already arrives faster than ROCm or oneAPI. Analysts pointed to Bright Computing as a precedent, which NVIDIA disputes. The GPL v2.0 licence permits forking. Observers named the speed of support for AMD's next chips as a concrete test.
Why it matters, evidence and limitations
- Why it matters
- Newly relevant (published April 7) because NVIDIA's Nov 2 Recovery release supports Slurm first and its Attribution tool ships Slurm connectors. Public universities and state HPC centres with mixed-vendor clusters should weigh how much operational tooling they tie to an NVIDIA-steered scheduler stack. The DOJ's broader NVIDIA review reportedly covers the SchedMD deal (Startup Fortune, Nov 2, not opened).
- Evidence and measured results
- Trade-press reporting with named analyst opinion and anonymous industry sources; no evidence of actual favouritism is presented.
- Limitations and uncertainty
- Opinion and anonymous sourcing; April article, so later Slurm releases may change the picture; NVIDIA disputes the Bright precedent.
How to read this edition
Source findings, measured results and limitations come from the cited publications. Patterns, operating questions, role takeaways and implementation considerations are Lighthouse Advisory interpretation, stated as questions to validate locally rather than guaranteed outcomes. Vendor and operator claims are labeled as claims. Full research method.
- Vendor claim
- A supplier-provided assertion that has not been upgraded to independent evidence.
- Academic research
- Research produced through an academic institution or peer-reviewed venue.
- Independent reporting
- Independent reporting with attributable sources but without a formal evaluation design.