{"resourceId":"nvidia-mission-control-2-5-autonomous-job-recovery-install-guide-20261026","versions":[{"version":"external-2e25ae3c3ae5fad97d0af82934cabc4c9b4abb89134d561506e6eaa463c43947","resource":{"id":"nvidia-mission-control-2-5-autonomous-job-recovery-install-guide-20261026","title":"NVIDIA Mission Control 2.5 install guide: autonomous job recovery runs in Slurm or Run:ai mode per installation, with Kubeflow and separate-cluster prerequisites","organization":"NVIDIA","sector":"AI factory and research-computing operations software","geography":"Global (product documentation)","publishedAt":"Version 2.5.0 documentation (no page date stated)","publicationDate":null,"eventDate":"2026-10-26","sourceName":"NVIDIA Mission Control Software Installation Guide 2.5.0","sourceLabel":"Vendor product documentation","sourceUrl":"https://docs.nvidia.com/mission-control/docs/nmc-software-installation-guide/2.5.0/are-install/are-install.html","evidenceClass":"vendor-claim","outcomeClass":"cautionary","topics":["infrastructure","governance-procurement"],"finding":"The 2.5.0 installation guide says autonomous job recovery ships in the on-premises bundle for DGX SuperPOD B200 and GB200 and supports one workload manager per installation: Slurm mode or Run:ai (Kubernetes) mode, mutually exclusive. Run:ai mode requires the Kubeflow Training Operator on the workload cluster, Run:ai already operational, and separate admin and workload Kubernetes clusters. The Run:ai restart controller ignores advice after three restarts by default, and the job recovery dashboard is not deployed or supported in Run:ai mode. A MySQL database is required.","sledRelevance":"Directly useful to campus and state research-computing teams evaluating whether the Oct 26 Kubernetes recovery announcement fits mixed Slurm and Kubernetes estates; it sets the practical scope.","evidence":"Vendor installation documentation read in the prerequisites, Run:ai integration and post-installation sections. It describes configuration and limits, not performance.","architectureImplications":"Admin Kubernetes cluster hosts core services; workload cluster runs Run:ai operators; Kubeflow PyTorchJob is the verified workload type. Mixed estates need separate installations.","governanceImplications":"Restart limits and advice handling are configurable policy; operators should document them so researchers know when automation stops retrying.","securityPrivacyImplications":"Installation creates database credentials and service accounts with cluster-level roles; credential handling and least privilege need review.","caveats":"Documentation, not evidence of outcomes. The page has no publication date; it is the 2.5.0 version tied to the Oct 26 release. Scope statements may change in later versions.","streamIds":["nvidia"],"roles":{"sales":"Interpretation: this guide is the reality check for the Oct 26 announcement. It lets account teams qualify early whether a university or state centre can use Kubernetes recovery as advertised: they need Run:ai, Kubeflow-based PyTorch training and a separate admin cluster, and they cannot cover Slurm and Run:ai in one installation. Raising these limits early protects credibility with research-computing directors, who will read the documentation themselves before agreeing to an upgrade.","engineering":"Interpretation: the guide sets concrete prerequisites and limits: mutually exclusive Slurm or Run:ai modes, the Kubeflow Training Operator, separate admin and workload clusters, MySQL, a default three-restart limit and no job recovery dashboard in Run:ai mode. Pre-sales should check whether the customer's training jobs use Kubeflow PyTorchJob, how many restarts long jobs tolerate, and whether their cluster design already separates admin and workload Kubernetes.","delivery":"Interpretation: delivery plans should include building or confirming the admin cluster, installing prerequisites in the right order (the Run:ai restart controller stays disabled until recovery is installed), setting restart limits and replacing the unsupported dashboard with Grafana views. Acceptance criteria should be proposed against documented behaviour, for example restarts stopping at the configured limit with a recorded warning event, and verified on a test PyTorch job."},"roleDetails":{"sales":{"customerProblem":"Research-computing centres need to know whether Kubernetes recovery fits their cluster design before committing to an upgrade.","stakeholders":["Research-computing director","Platform engineering lead","Procurement or licensing office"],"discoveryQuestions":["Do your AI teams submit Kubeflow PyTorchJob workloads through Run:ai?","Do you run Slurm and Kubernetes on the same SuperPOD?","Do you already operate a separate admin Kubernetes cluster?"],"valueHypothesis":"Interpretation: qualifying fit early avoids failed upgrade projects; the guide gives no performance benefit.","potentialEngagement":"Short technical fit review of prerequisites against the customer's current cluster design.","unsupportedClaims":["A single installation covering Slurm and Run:ai","Any workload type beyond those documented","Performance improvements"]},"engineering":{"fit":"DGX SuperPOD B200 or GB200 customers with Run:ai and Kubeflow-based PyTorch training.","architectureIntegration":"Core recovery services on an admin Kubernetes cluster; Run:ai operators and advice restart controller on the workload cluster; MySQL backing store.","prerequisites":["Kubeflow Training Operator","Operational Run:ai","Separate admin and workload clusters","MySQL database"],"constraints":["Slurm and Run:ai modes are mutually exclusive","Default restart limit of three","Job recovery dashboard unsupported in Run:ai mode"],"security":"Review service accounts, database credentials and cluster roles created at install.","proofOfValue":["Submit a test PyTorchJob and confirm advice and restart events","Confirm restarts stop at the configured limit with a warning event"]},"delivery":{"work":"Prepare clusters and prerequisites, install in Run:ai mode, configure the restart controller and Grafana dashboards.","dependencies":["Run:ai","Kubeflow","BCM tooling","MySQL"],"ownership":"Platform engineering owns installation and configuration.","skillsAdoption":"Operators need BCM, Kubernetes and Run:ai administration skills.","governanceCheckpoints":["Change approval for restart-limit settings","Security review of install-time credentials"],"acceptanceCriteria":["Proposed: a test PyTorchJob restarts automatically after an injected failure and stops at the configured limit","Proposed: Grafana dashboards show recovery events for the test"],"risks":["Prerequisite ordering mistakes","Monitoring gaps from the unsupported dashboard","Scope changes in later releases"]}},"retrievedAt":"2026-10-27T02:56:53Z","enrichedAt":"2026-10-27T03:07:35Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Installation assumes BCM, Kubernetes and Run:ai administration skills.","procurementImplications":"Buyers should confirm that Run:ai licensing and Kubeflow are in scope and budget for operating two installations if they need recovery on both schedulers.","operatingModelImplications":"Recovery automation is per scheduler, so centres serving HPC and AI communities on different schedulers run parallel operating processes.","sourceVerification":{"openedUrl":"https://docs.nvidia.com/mission-control/docs/nmc-software-installation-guide/2.5.0/are-install/are-install.html","referenceExcerpt":"You can deploy Mission Control autonomous job recovery to a Slurm cluster or to a Run:ai cluster, but not both in the same installation.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}