From the NVIDIA edition of October 26, 2026
NVIDIA Mission Control 2.5 install guide: autonomous job recovery runs in Slurm or Run:ai mode per installation, with Kubeflow and separate-cluster prerequisites
NVIDIA · AI factory and research-computing operations software · Global (product documentation)
- Publisher
- NVIDIA Mission Control Software Installation Guide 2.5.0
- Original publication
- Version 2.5.0 documentation (no page date stated)
- Source retrieved
- 2026-10-27
- Event date
- 2026-10-26
What happened
The 2.5.0 installation guide says autonomous job recovery ships in the on-premises bundle for DGX SuperPOD B200 and GB200 and supports one workload manager per installation: Slurm mode or Run:ai (Kubernetes) mode, mutually exclusive. Run:ai mode requires the Kubeflow Training Operator on the workload cluster, Run:ai already operational, and separate admin and workload Kubernetes clusters. The Run:ai restart controller ignores advice after three restarts by default, and the job recovery dashboard is not deployed or supported in Run:ai mode. A MySQL database is required.
Why it matters
Directly useful to campus and state research-computing teams evaluating whether the Oct 26 Kubernetes recovery announcement fits mixed Slurm and Kubernetes estates; it sets the practical scope.
Evidence and measured results
Vendor installation documentation read in the prerequisites, Run:ai integration and post-installation sections. It describes configuration and limits, not performance.
Limitations and uncertainty
Documentation, not evidence of outcomes. The page has no publication date; it is the 2.5.0 version tied to the Oct 26 release. Scope statements may change in later versions.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-27; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Research-computing centres need to know whether Kubernetes recovery fits their cluster design before committing to an upgrade.
- Stakeholders
- Research-computing director
- Platform engineering lead
- Procurement or licensing office
- Discovery questions
- Do your AI teams submit Kubeflow PyTorchJob workloads through Run:ai?
- Do you run Slurm and Kubernetes on the same SuperPOD?
- Do you already operate a separate admin Kubernetes cluster?
- Value hypothesis
- Interpretation: qualifying fit early avoids failed upgrade projects; the guide gives no performance benefit.
- Potential engagement
- Short technical fit review of prerequisites against the customer's current cluster design.
- Unsupported claims
- A single installation covering Slurm and Run:ai
- Any workload type beyond those documented
- Performance improvements
Pre-sales engineering
Role takeaway
- Fit
- DGX SuperPOD B200 or GB200 customers with Run:ai and Kubeflow-based PyTorch training.
- Architecture and integration
- Core recovery services on an admin Kubernetes cluster; Run:ai operators and advice restart controller on the workload cluster; MySQL backing store.
- Prerequisites
- Kubeflow Training Operator
- Operational Run:ai
- Separate admin and workload clusters
- MySQL database
- Constraints
- Slurm and Run:ai modes are mutually exclusive
- Default restart limit of three
- Job recovery dashboard unsupported in Run:ai mode
- Security
- Review service accounts, database credentials and cluster roles created at install.
- Proof of value
- Submit a test PyTorchJob and confirm advice and restart events
- Confirm restarts stop at the configured limit with a warning event
Delivery
Role takeaway
- Work
- Prepare clusters and prerequisites, install in Run:ai mode, configure the restart controller and Grafana dashboards.
- Dependencies
- Run:ai
- Kubeflow
- BCM tooling
- MySQL
- Ownership
- Platform engineering owns installation and configuration.
- Skills and adoption
- Operators need BCM, Kubernetes and Run:ai administration skills.
- Governance checkpoints
- Change approval for restart-limit settings
- Security review of install-time credentials
- Acceptance criteria
- Proposed: a test PyTorchJob restarts automatically after an injected failure and stops at the configured limit
- Proposed: Grafana dashboards show recovery events for the test
- Risks
- Prerequisite ordering mistakes
- Monitoring gaps from the unsupported dashboard
- Scope changes in later releases
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Admin Kubernetes cluster hosts core services; workload cluster runs Run:ai operators; Kubeflow PyTorchJob is the verified workload type. Mixed estates need separate installations.
Governance
Who approves, reviews and stays accountable for outcomes?
Restart limits and advice handling are configurable policy; operators should document them so researchers know when automation stops retrying.
Security and privacy
What data, permissions and controls need testing?
Installation creates database credentials and service accounts with cluster-level roles; credential handling and least privilege need review.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Installation assumes BCM, Kubernetes and Run:ai administration skills.
Procurement
What should contracts, pricing and exit terms secure?
Buyers should confirm that Run:ai licensing and Kubeflow are in scope and budget for operating two installations if they need recovery on both schedulers.
Operating model
Which teams own the service once it runs?
Recovery automation is per scheduler, so centres serving HPC and AI communities on different schedulers run parallel operating processes.
Publication history
- 2026-10-26NVIDIA · Issue 518 resources
Stable resource ID: nvidia-mission-control-2-5-autonomous-job-recovery-install-guide-20261026