From the NVIDIA edition of November 2, 2026
NVIDIA open-sources Recovery, a cluster-level service that detects, isolates and restarts failed training jobs, starting with Slurm
NVIDIA · AI infrastructure operations software · Global
- Publisher
- NVIDIA Technical Blog
- Original publication
- 2026-11-02
- Source retrieved
- 2026-11-03
- Event date
- 2026-11-02
What happened
NVIDIA released NVIDIA Recovery, part of its DSX OS software, under Apache 2.0. It runs a four-stage loop (detect, attribute, isolate, restart or stop) across a cluster's training jobs, using application logs, GPU metrics and scheduler signals. It supports Slurm today through a Helm-deployed Advisor Controller (in-cluster or SSH mode); Kubernetes support is on the roadmap. New deployments start in monitor mode, recording what they would have done without acting, and every action goes to a persistent audit trail. NVIDIA reports that on one training cluster from July 9 to October 9 it covered nearly 200 jobs per day, logged about 8,550 anomaly detections, and had mean detection and recovery-action latencies of 69 and 94 seconds. NVIDIA says IBM Research ran it in monitor mode before enabling autonomous action, and neocloud Verda contributed a VictoriaLogs connector.
Why it matters
University research computing centres and state HPC facilities that run Slurm for shared GPU clusters lose GPU-hours when multi-node jobs fail and staff restart them by hand. This is a free, open-source option they can trial in observe-only mode. Relevance is limited to sites running large distributed training, not most agency AI use.
Evidence and measured results
Vendor blog with one production observation window on a single NVIDIA cluster (detections and latencies only). No figures on false positives, recovery success rate or goodput gained; the IBM Research and Verda mentions are vendor-reported.
Limitations and uncertainty
NVIDIA's own metrics; recovery-action latency ends at the action, not when training resumes. Slurm only for now. No independent evaluation.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-03; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Research-computing centres lose GPU-hours and staff time to manual restarts of failed multi-node training jobs.
- Stakeholders
- University research computing director
- HPC operations lead
- Research faculty running large training jobs
- Discovery questions
- How many multi-node jobs fail per week, and who restarts them?
- Do your jobs checkpoint regularly?
- Are you on Slurm, Kubernetes, or both?
- Value hypothesis
- An observe-only trial can show whether automated recommendations match staff decisions, at no licence cost.
- Potential engagement
- A short assessment of failure handling plus a monitor-mode pilot plan on one Slurm cluster.
- Unsupported claims
- GPU-hour or cost savings
- Recovery success rates
- Kubernetes support today
- Performance on non-NVIDIA clusters
Pre-sales engineering
Role takeaway
- Fit
- Fits Slurm sites running distributed training with existing log and metric pipelines.
- Architecture and integration
- Control-plane service calling a Slurm Advisor Controller via HTTP/gRPC; ingests Loki/VictoriaLogs and Prometheus; MySQL state store.
- Prerequisites
- Slurm cluster
- Loki or VictoriaLogs
- Prometheus
- Managed MySQL for production
- Checkpointing jobs
- Constraints
- Slurm only; Kubernetes on roadmap
- Metrics from one NVIDIA cluster
- Recovery latency excludes checkpoint restore time
- Security
- SSH mode grants remote scheduler control; centralised job logs need access controls and retention rules.
- Proof of value
- Run monitor mode and count false positives and misses against operator decisions
- Measure time from failure to resumed training, not just action latency
Delivery
Role takeaway
- Work
- Pilot NVIDIA Recovery in monitor mode, validate signals, then enable action mode by cluster.
- Dependencies
- Telemetry stack
- Slurm administrators
- Job checkpointing by users
- Ownership
- Research-computing operations team.
- Skills and adoption
- Operators learn the policy engine and audit trail; users learn checkpoint practice.
- Governance checkpoints
- Security review of SSH or in-cluster controller
- Accuracy review before action mode
- Periodic audit-trail review
- Acceptance criteria
- Proposed: at least 90% of monitor-mode recommendations agree with operator decisions over four weeks
- Proposed: no wrongful job kills in the first month of action mode
- Risks
- Wrongful kills or requeues
- Incomplete telemetry
- Vendor-led roadmap
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Control-plane service with HTTP/gRPC Advisor API; needs Loki or VictoriaLogs, Prometheus and a MySQL database; jobs must already checkpoint and restart.
Governance
Who approves, reviews and stays accountable for outcomes?
Monitor-mode default and an audit trail support a staged approval before letting software kill or requeue user jobs.
Security and privacy
What data, permissions and controls need testing?
SSH mode gives a remote control plane scheduler access; job logs are ingested centrally and need access controls.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Reduces after-hours manual recovery work for small research-computing teams, if signals are accurate.
Procurement
What should contracts, pricing and exit terms secure?
Open source, so no licence cost, but support and integration effort are not covered.
Operating model
Which teams own the service once it runs?
Shifts recovery from individual users to a cluster-level policy owned by the operations team.
Publication history
- 2026-11-02NVIDIA · Issue 585 resources
Stable resource ID: nvidia-recovery-open-source-cluster-workload-recovery-slurm-20261102