{"resourceId":"nvidia-recovery-open-source-cluster-workload-recovery-slurm-20261102","versions":[{"version":"external-e1c4c959232b1a856a746defcf560b286b9241c159f8688e8ba5e3e08793e2e3","resource":{"id":"nvidia-recovery-open-source-cluster-workload-recovery-slurm-20261102","title":"NVIDIA open-sources Recovery, a cluster-level service that detects, isolates and restarts failed training jobs, starting with Slurm","organization":"NVIDIA","sector":"AI infrastructure operations software","geography":"Global","publishedAt":"2026-11-02","publicationDate":"2026-11-02","eventDate":"2026-11-02","sourceName":"NVIDIA Technical Blog","sourceLabel":"NVIDIA blog: Keep Distributed Training Running with Automated Workload Recovery","sourceUrl":"https://developer.nvidia.com/blog/keep-distributed-training-running-with-automated-workload-recovery/","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["infrastructure","operating-model"],"finding":"NVIDIA released NVIDIA Recovery, part of its DSX OS software, under Apache 2.0. It runs a four-stage loop (detect, attribute, isolate, restart or stop) across a cluster's training jobs, using application logs, GPU metrics and scheduler signals. It supports Slurm today through a Helm-deployed Advisor Controller (in-cluster or SSH mode); Kubernetes support is on the roadmap. New deployments start in monitor mode, recording what they would have done without acting, and every action goes to a persistent audit trail. NVIDIA reports that on one training cluster from July 9 to October 9 it covered nearly 200 jobs per day, logged about 8,550 anomaly detections, and had mean detection and recovery-action latencies of 69 and 94 seconds. NVIDIA says IBM Research ran it in monitor mode before enabling autonomous action, and neocloud Verda contributed a VictoriaLogs connector.","sledRelevance":"University research computing centres and state HPC facilities that run Slurm for shared GPU clusters lose GPU-hours when multi-node jobs fail and staff restart them by hand. This is a free, open-source option they can trial in observe-only mode. Relevance is limited to sites running large distributed training, not most agency AI use.","evidence":"Vendor blog with one production observation window on a single NVIDIA cluster (detections and latencies only). No figures on false positives, recovery success rate or goodput gained; the IBM Research and Verda mentions are vendor-reported.","architectureImplications":"Control-plane service with HTTP/gRPC Advisor API; needs Loki or VictoriaLogs, Prometheus and a MySQL database; jobs must already checkpoint and restart.","governanceImplications":"Monitor-mode default and an audit trail support a staged approval before letting software kill or requeue user jobs.","securityPrivacyImplications":"SSH mode gives a remote control plane scheduler access; job logs are ingested centrally and need access controls.","caveats":"NVIDIA's own metrics; recovery-action latency ends at the action, not when training resumes. Slurm only for now. No independent evaluation.","streamIds":["nvidia"],"roles":{"sales":"Position this as a low-risk pilot for research-computing clients, not a cost-savings promise. NVIDIA Recovery is free and starts in monitor mode, so a university HPC centre can compare its advice with what staff actually did before allowing any automatic action. The only figures are NVIDIA's own detection and latency numbers from one cluster. Offer a bounded assessment of current failure and restart practices and a monitor-mode trial plan. Do not claim goodput or GPU-hour savings; the source does not measure them, and the tool only helps sites running multi-node training on Slurm.","engineering":"Treat this as an integration project around existing telemetry. Recovery needs Loki or VictoriaLogs, Prometheus, a MySQL database and a Slurm Advisor Controller deployed by Helm, in-cluster or over SSH. Kubernetes is not yet supported. Jobs must already save checkpoints and restart from them. Use the log ingestion configurator to test signals before the first workload. For a proof of value, run monitor mode for several weeks and score each recommendation against operator decisions, counting false positives and missed failures. Review SSH-mode access and log handling with security before any production use.","delivery":"Deliver this in stages with explicit gates. Start in monitor mode, have operators label each recommendation, and switch a cluster to action mode only after an agreed accuracy threshold. The research-computing operations team owns policy, node-exclusion rules and the audit trail. Users need guidance on checkpoint frequency so restarts actually save work. Track recovery-action latency separately from time to resumed training, since NVIDIA notes they differ. Risks include wrongful job kills, telemetry gaps, and dependence on an NVIDIA-led roadmap for a scheduler that NVIDIA also now owns through SchedMD."},"roleDetails":{"sales":{"customerProblem":"Research-computing centres lose GPU-hours and staff time to manual restarts of failed multi-node training jobs.","stakeholders":["University research computing director","HPC operations lead","Research faculty running large training jobs"],"discoveryQuestions":["How many multi-node jobs fail per week, and who restarts them?","Do your jobs checkpoint regularly?","Are you on Slurm, Kubernetes, or both?"],"valueHypothesis":"An observe-only trial can show whether automated recommendations match staff decisions, at no licence cost.","potentialEngagement":"A short assessment of failure handling plus a monitor-mode pilot plan on one Slurm cluster.","unsupportedClaims":["GPU-hour or cost savings","Recovery success rates","Kubernetes support today","Performance on non-NVIDIA clusters"]},"engineering":{"fit":"Fits Slurm sites running distributed training with existing log and metric pipelines.","architectureIntegration":"Control-plane service calling a Slurm Advisor Controller via HTTP/gRPC; ingests Loki/VictoriaLogs and Prometheus; MySQL state store.","prerequisites":["Slurm cluster","Loki or VictoriaLogs","Prometheus","Managed MySQL for production","Checkpointing jobs"],"constraints":["Slurm only; Kubernetes on roadmap","Metrics from one NVIDIA cluster","Recovery latency excludes checkpoint restore time"],"security":"SSH mode grants remote scheduler control; centralised job logs need access controls and retention rules.","proofOfValue":["Run monitor mode and count false positives and misses against operator decisions","Measure time from failure to resumed training, not just action latency"]},"delivery":{"work":"Pilot NVIDIA Recovery in monitor mode, validate signals, then enable action mode by cluster.","dependencies":["Telemetry stack","Slurm administrators","Job checkpointing by users"],"ownership":"Research-computing operations team.","skillsAdoption":"Operators learn the policy engine and audit trail; users learn checkpoint practice.","governanceCheckpoints":["Security review of SSH or in-cluster controller","Accuracy review before action mode","Periodic audit-trail review"],"acceptanceCriteria":["Proposed: at least 90% of monitor-mode recommendations agree with operator decisions over four weeks","Proposed: no wrongful job kills in the first month of action mode"],"risks":["Wrongful kills or requeues","Incomplete telemetry","Vendor-led roadmap"]}},"retrievedAt":"2026-11-03T04:00:34Z","enrichedAt":"2026-11-03T04:04:41Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Reduces after-hours manual recovery work for small research-computing teams, if signals are accurate.","procurementImplications":"Open source, so no licence cost, but support and integration effort are not covered.","operatingModelImplications":"Shifts recovery from individual users to a cluster-level policy owned by the operations team.","sourceVerification":{"openedUrl":"https://developer.nvidia.com/blog/keep-distributed-training-running-with-automated-workload-recovery/","referenceExcerpt":"Recovery-action latency ends at the automated action; scheduling and checkpoint restoration determine when productive training resumes.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}