{"resourceId":"nvidia-mission-control-2-5-kubernetes-autonomous-recovery-20261026","versions":[{"version":"external-2e25ae3c3ae5fad97d0af82934cabc4c9b4abb89134d561506e6eaa463c43947","resource":{"id":"nvidia-mission-control-2-5-kubernetes-autonomous-recovery-20261026","title":"NVIDIA (Oct 26): Mission Control 2.5 extends autonomous job and hardware recovery from Slurm to Kubernetes workloads scheduled by Run:ai","organization":"NVIDIA","sector":"AI factory and research-computing operations software","geography":"Global (product release)","publishedAt":"2026-10-26 (NVIDIA Technical Blog)","publicationDate":"2026-10-26","eventDate":"2026-10-26","sourceName":"NVIDIA Technical Blog","sourceLabel":"Vendor technical blog","sourceUrl":"https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes/","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["infrastructure","operating-model"],"finding":"NVIDIA says Mission Control 2.5, available Oct 26, extends its autonomous recovery engine, previously limited to Slurm-scheduled HPC workloads, to Kubernetes workloads managed by NVIDIA Run:ai. The engine is two services: autonomous job recovery, which classifies failures using job context and history and drives restart, retry or requeue through the scheduler, and autonomous hardware recovery, which isolates suspect nodes, records the reason and runs diagnostics in parallel. The post walks through three scenarios (hardware fault, transient software fault, early degradation). Other 2.5 items are a beta AI network anomaly detection service, central Keycloak authentication, Kubernetes hardening and early access for Vera Rubin NVL72. No measured recovery times, uptime or utilisation results are given.","sledRelevance":"Relevant to universities and state research-computing centres that run shared DGX SuperPOD clusters with both Slurm (traditional HPC users) and Kubernetes (AI and ML teams). Limited relevance for agencies without on-premises SuperPOD-class systems.","evidence":"Vendor announcement blog with architecture diagrams and scenario walkthroughs. No customer, benchmark, baseline or failure statistics are reported; benefits are described qualitatively.","architectureImplications":"Three-layer control loop (scheduler, job recovery, hardware recovery). Run:ai is the default Kubernetes scheduler within Mission Control. Installation documentation (filed separately) shows a separate admin Kubernetes cluster and one workload manager per installation.","governanceImplications":"Automated node quarantine and job restarts change who decides when shared capacity is taken out of service; research-computing centres need policies for automated actions and user notification.","securityPrivacyImplications":"The release lists central Keycloak authentication and Kubernetes hardening but gives no detail; recovery services need privileged access to the cluster and job logs.","caveats":"Vendor blog with no measured outcomes. The installation guides show narrower scope than the post implies: one workload manager per installation, Kubeflow Training Operator required for Run:ai mode, and hardware health checks and Break/Fix limited to GB200.","streamIds":["nvidia"],"roles":{"sales":"Interpretation: Mission Control 2.5 gives a talking point for universities and state research-computing centres that bought SuperPOD-class systems and now serve AI teams on Kubernetes alongside Slurm users. The honest pitch is reduced manual triage, not a quantified uptime gain, because NVIDIA reports no measured results. Discovery should establish current failure rates, time lost to stalled jobs and which scheduler each user community uses, and should flag the GB200-only hardware diagnostics before anyone with GB300 or B200 expects the full feature set.","engineering":"Interpretation: the design separates workload recovery from hardware remediation, which is sound, but the installation guide narrows fit. Each installation supports Slurm or Run:ai, not both; Run:ai mode needs the Kubeflow Training Operator and separate admin and workload clusters; the job recovery dashboard is not supported in Run:ai mode; and health checks and Break/Fix are GB200-only. Pre-sales should map these against the customer's hardware generation and job types before proposing it, and test restart behaviour on real PyTorch jobs.","delivery":"Interpretation: delivery is an upgrade and integration project, not a switch-on. Teams need to plan the admin cluster, Run:ai and Kubeflow prerequisites, restart limits and alert routing, and agree a policy for automated node quarantine with researchers. Acceptance should be proposed as measurable changes against a pre-upgrade baseline, such as time from fault to job resumption and share of false quarantines, measured over a defined period, rather than relying on NVIDIA's qualitative description."},"roleDetails":{"sales":{"customerProblem":"Shared university or state research clusters lose GPU hours when failed jobs and faulty nodes wait for manual triage, especially as AI teams move to Kubernetes.","stakeholders":["Research-computing director","HPC and AI platform engineers","CIO or vice president for research","Principal investigators with large training jobs"],"discoveryQuestions":["Which share of your GPU workload runs on Slurm and which on Kubernetes or Run:ai?","How many job failures per month need manual triage, and how long do they take?","Which DGX generation (B200, GB200, GB300) do you run?","Who approves taking a node out of service today?"],"valueHypothesis":"Interpretation: automated restart and node isolation may reduce manual triage time on shared clusters; NVIDIA reports no measured size of benefit.","potentialEngagement":"Bounded readiness assessment of Mission Control 2.5 recovery features against the customer's scheduler mix and hardware generation, with a baseline of current failure handling.","unsupportedClaims":["Any quantified uptime, MTTR or utilisation gain (none reported)","Full hardware diagnostics on GB300 or B200 (the guide limits them to GB200)","Running Slurm and Run:ai recovery in one installation"]},"engineering":{"fit":"SuperPOD-class on-premises clusters running Run:ai-scheduled PyTorch training on Kubernetes, ideally GB200.","architectureIntegration":"Run:ai scheduler, autonomous job recovery services on a separate admin Kubernetes cluster, autonomous hardware recovery agents on compute nodes, Grafana dashboards.","prerequisites":["Mission Control 2.5 upgrade path","Run:ai installed on the workload cluster","Kubeflow Training Operator","Separate admin Kubernetes cluster"],"constraints":["One workload manager per installation","Hardware health checks and Break/Fix GB200-only","Job recovery dashboard unsupported in Run:ai mode","No published performance data"],"security":"Recovery services need privileged cluster and log access; confirm Keycloak integration with campus identity and review the stated Kubernetes hardening.","proofOfValue":["Inject a controlled GPU fault and time detection to job resumption","Run transient failures and confirm no hardware quarantine","Check restart-limit behaviour on long PyTorch jobs"]},"delivery":{"work":"Plan and run the 2.5 upgrade, install recovery services in the chosen mode, configure restart limits and alerting, and document automated-action policy.","dependencies":["Run:ai licence and installation","Kubeflow Training Operator","NVIDIA support for upgrade","Researcher communication channels"],"ownership":"The research-computing operations team owns the recovery services and quarantine policy.","skillsAdoption":"Operators need Kubernetes and Run:ai skills; researchers need guidance on how restarted jobs and checkpoints behave.","governanceCheckpoints":["Approve automated quarantine and restart policy","Confirm supported features for the installed hardware generation","Review access granted to recovery services"],"acceptanceCriteria":["Proposed: median time from injected fault to job resumption measured and compared with a pre-upgrade baseline","Proposed: no node quarantined for a test transient software fault"],"risks":["Expecting GB200 features on other hardware","Automation restarting jobs that should stop","Operating two installations for mixed estates"]}},"retrievedAt":"2026-10-27T02:56:52Z","enrichedAt":"2026-10-27T03:07:35Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Shifts cluster operations staff from manual triage toward reviewing automated actions; still requires Kubernetes, Run:ai and Slurm skills.","procurementImplications":"Mission Control is bundled with DGX SuperPOD on-premises software; buyers should confirm which recovery features are supported for their exact system generation.","operatingModelImplications":"Moves shared clusters toward automated, scheduler-aware recovery; centres running both Slurm and Kubernetes still need two installations under the documented model.","sourceVerification":{"openedUrl":"https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes/","referenceExcerpt":"The autonomous job recovery service doesn’t need to repair hardware and the hardware recovery service doesn’t need to understand every training framework.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}