From the NVIDIA edition of October 26, 2026
NVIDIA (Oct 26): Mission Control 2.5 extends autonomous job and hardware recovery from Slurm to Kubernetes workloads scheduled by Run:ai
NVIDIA · AI factory and research-computing operations software · Global (product release)
- Publisher
- NVIDIA Technical Blog
- Original publication
- 2026-10-26 (NVIDIA Technical Blog)
- Source retrieved
- 2026-10-27
- Event date
- 2026-10-26
What happened
NVIDIA says Mission Control 2.5, available Oct 26, extends its autonomous recovery engine, previously limited to Slurm-scheduled HPC workloads, to Kubernetes workloads managed by NVIDIA Run:ai. The engine is two services: autonomous job recovery, which classifies failures using job context and history and drives restart, retry or requeue through the scheduler, and autonomous hardware recovery, which isolates suspect nodes, records the reason and runs diagnostics in parallel. The post walks through three scenarios (hardware fault, transient software fault, early degradation). Other 2.5 items are a beta AI network anomaly detection service, central Keycloak authentication, Kubernetes hardening and early access for Vera Rubin NVL72. No measured recovery times, uptime or utilisation results are given.
Why it matters
Relevant to universities and state research-computing centres that run shared DGX SuperPOD clusters with both Slurm (traditional HPC users) and Kubernetes (AI and ML teams). Limited relevance for agencies without on-premises SuperPOD-class systems.
Evidence and measured results
Vendor announcement blog with architecture diagrams and scenario walkthroughs. No customer, benchmark, baseline or failure statistics are reported; benefits are described qualitatively.
Limitations and uncertainty
Vendor blog with no measured outcomes. The installation guides show narrower scope than the post implies: one workload manager per installation, Kubeflow Training Operator required for Run:ai mode, and hardware health checks and Break/Fix limited to GB200.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-27; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Shared university or state research clusters lose GPU hours when failed jobs and faulty nodes wait for manual triage, especially as AI teams move to Kubernetes.
- Stakeholders
- Research-computing director
- HPC and AI platform engineers
- CIO or vice president for research
- Principal investigators with large training jobs
- Discovery questions
- Which share of your GPU workload runs on Slurm and which on Kubernetes or Run:ai?
- How many job failures per month need manual triage, and how long do they take?
- Which DGX generation (B200, GB200, GB300) do you run?
- Who approves taking a node out of service today?
- Value hypothesis
- Interpretation: automated restart and node isolation may reduce manual triage time on shared clusters; NVIDIA reports no measured size of benefit.
- Potential engagement
- Bounded readiness assessment of Mission Control 2.5 recovery features against the customer's scheduler mix and hardware generation, with a baseline of current failure handling.
- Unsupported claims
- Any quantified uptime, MTTR or utilisation gain (none reported)
- Full hardware diagnostics on GB300 or B200 (the guide limits them to GB200)
- Running Slurm and Run:ai recovery in one installation
Pre-sales engineering
Role takeaway
- Fit
- SuperPOD-class on-premises clusters running Run:ai-scheduled PyTorch training on Kubernetes, ideally GB200.
- Architecture and integration
- Run:ai scheduler, autonomous job recovery services on a separate admin Kubernetes cluster, autonomous hardware recovery agents on compute nodes, Grafana dashboards.
- Prerequisites
- Mission Control 2.5 upgrade path
- Run:ai installed on the workload cluster
- Kubeflow Training Operator
- Separate admin Kubernetes cluster
- Constraints
- One workload manager per installation
- Hardware health checks and Break/Fix GB200-only
- Job recovery dashboard unsupported in Run:ai mode
- No published performance data
- Security
- Recovery services need privileged cluster and log access; confirm Keycloak integration with campus identity and review the stated Kubernetes hardening.
- Proof of value
- Inject a controlled GPU fault and time detection to job resumption
- Run transient failures and confirm no hardware quarantine
- Check restart-limit behaviour on long PyTorch jobs
Delivery
Role takeaway
- Work
- Plan and run the 2.5 upgrade, install recovery services in the chosen mode, configure restart limits and alerting, and document automated-action policy.
- Dependencies
- Run:ai licence and installation
- Kubeflow Training Operator
- NVIDIA support for upgrade
- Researcher communication channels
- Ownership
- The research-computing operations team owns the recovery services and quarantine policy.
- Skills and adoption
- Operators need Kubernetes and Run:ai skills; researchers need guidance on how restarted jobs and checkpoints behave.
- Governance checkpoints
- Approve automated quarantine and restart policy
- Confirm supported features for the installed hardware generation
- Review access granted to recovery services
- Acceptance criteria
- Proposed: median time from injected fault to job resumption measured and compared with a pre-upgrade baseline
- Proposed: no node quarantined for a test transient software fault
- Risks
- Expecting GB200 features on other hardware
- Automation restarting jobs that should stop
- Operating two installations for mixed estates
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Three-layer control loop (scheduler, job recovery, hardware recovery). Run:ai is the default Kubernetes scheduler within Mission Control. Installation documentation (filed separately) shows a separate admin Kubernetes cluster and one workload manager per installation.
Governance
Who approves, reviews and stays accountable for outcomes?
Automated node quarantine and job restarts change who decides when shared capacity is taken out of service; research-computing centres need policies for automated actions and user notification.
Security and privacy
What data, permissions and controls need testing?
The release lists central Keycloak authentication and Kubernetes hardening but gives no detail; recovery services need privileged access to the cluster and job logs.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Shifts cluster operations staff from manual triage toward reviewing automated actions; still requires Kubernetes, Run:ai and Slurm skills.
Procurement
What should contracts, pricing and exit terms secure?
Mission Control is bundled with DGX SuperPOD on-premises software; buyers should confirm which recovery features are supported for their exact system generation.
Operating model
Which teams own the service once it runs?
Moves shared clusters toward automated, scheduler-aware recovery; centres running both Slurm and Kubernetes still need two installations under the documented model.
Publication history
- 2026-10-26NVIDIA · Issue 518 resources
Stable resource ID: nvidia-mission-control-2-5-kubernetes-autonomous-recovery-20261026