{"resourceId":"nvidia-nodewright-oss-20260923","versions":[{"version":"external-a773ab94e74681c35080945cb66d6ea5beab0f76a39e43b49a61d6b378069255","resource":{"id":"nvidia-nodewright-oss-20260923","title":"NodeWright: declarative host-OS fleet updates for GPU Kubernetes without killing training","organization":"NVIDIA","sector":"AI factory / Kubernetes host operations","geography":"Global (vendor OSS; U.S. SLED transferable with ops limits)","publishedAt":"2026-09-23T00:00:00Z","publicationDate":"2026-09-23","eventDate":"2026-09-23","sourceName":"NVIDIA Developer Blog","sourceLabel":"Vendor technical blog","sourceUrl":"https://developer.nvidia.com/blog/manage-kubernetes-node-fleets-with-nodewright/","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["infrastructure","operating-model","developers-agents"],"finding":"On Sep 23, 2026 NVIDIA introduced NodeWright (formerly internal Skyhook) as an Apache-2.0, Kubernetes-native package manager inside DSX OS that declaratively configures and updates host OS packages across GPU node fleets. The operator cordons, waits for non-interruptible pods, drains honoring PodDisruptionBudgets, applies packages, optionally interrupts/reboots, then uncordons. DeploymentPolicy supports fixed/linear/exponential progressive rollouts with success and failure thresholds. Packages cover sysctl/GRUB, CVE remediation, security agents, crash dumps, and accelerator-intent tuning (Hopper/Blackwell; GKE COS variants). NodeWright integrates with AI Cluster Runtime recipes and is positioned beside NVCRE (pre-workload validation) and NVSentinel (runtime faults); it does not replace GPU Operator or Network Operator.","sledRelevance":"Campus and agency GPU Kubernetes fleets that cannot discard scarce GPU nodes for host CVEs/kernel/RDMA changes can evaluate NodeWright-style disruption-aware rollouts without inventing a new SLED application use case.","evidence":"Blog dated Sep 23, 2026; describes six-stage sequence, DeploymentPolicy strategies, package validation checks, new-node taint-until-validated path, Helm OCI install, and DSX OS / AICR / NVCRE / NVSentinel framing.","architectureImplications":"Treat host OS change as a CRD-driven fleet workflow under GPU Operator; separate host packages from device-plugin/driver ownership; plan progressive compartments by nodepool labels.","governanceImplications":"Who approves interruptionBudget percent, non-interruptible workload labels, and failure-threshold stops before production rollouts?","securityPrivacyImplications":"Packages run root-level host changes; RBAC, admission control, and package provenance become part of the trust boundary.","caveats":"Vendor introduction; no independent campus measured MTTD/MTTR for NodeWright rollouts in this post; Skyhook production claims are NVIDIA-internal.","streamIds":["nvidia"],"roles":{"sales":"Frame NodeWright as safe host-fleet change control for scarce GPU nodes—not a replacement for GPU Operator. Open with the buyer’s 3 a.m. maintenance-window pain, then cite Sep 23 DSX OS packaging: cordon/drain/PDB-aware packages and progressive DeploymentPolicy. Bound the ask to a canary compartment on one labeled nodepool with a non-destructive package (sysctl/validation) before CVE remediations.","engineering":"Fit is strong where Ansible/Puppet ignore Kubernetes disruption budgets on GPU workers; weak if the cluster still lacks GPU Operator/driver lifecycle. Integrate NodeWright CRs via GitOps; keep GPU Operator owning drivers/toolkit. Prerequisites: Kubernetes cluster, Helm OCI chart, node labels, PDBs. Constraints: host root packages; progressive rollout still burns capacity during drains. Proof: package status/conditions and stopped rollouts on failed validation checks.","delivery":"Work: install operator, define packages and DeploymentPolicy compartments, label non-interruptible training pods, run canary then expand. Dependencies: GitOps, node labeling, package image registry, on-call for failed Jobs. Ownership: platform SRE owning host config. Skills: Kubernetes CRDs, host packaging, GPU job drain discipline. Governance: change board for interruption budgets. Acceptance: canary success threshold met; zero unintended training kills. Risks: bad package bricks nodes; over-wide selectors."},"roleDetails":{"sales":{"customerProblem":"Host CVEs and kernel/RDMA changes force brittle GPU maintenance windows that risk killing multi-day training.","stakeholders":["platform SRE","HPC/AI ops","security/compliance","research computing"],"discoveryQuestions":["How do you update GPU node hosts today without recycling hardware?","Which workloads are labeled non-interruptible?","What success/failure thresholds would stop a fleet rollout?"],"valueHypothesis":"Disruption-aware declarative host packages can reduce emergency windows if budgets and labels match real jobs.","potentialEngagement":"Pilot NodeWright canary on one gpu nodepool with a validation-only package.","unsupportedClaims":["NodeWright replaces GPU Operator","Open-sourcing Skyhook proves every SLED cluster is production-ready","Progressive rollouts eliminate all training risk"]},"engineering":{"fit":"Strong for K8s GPU fleets needing host-level change control; weak as a driver/toolkit substitute.","architectureIntegration":"Layer NodeWright under GPU Operator/Network Operator; consume AICR-rendered package pins when used.","prerequisites":["Kubernetes","Helm OCI install","node labels/selectors","PodDisruptionBudgets"],"constraints":["Root-capable packages","Drain capacity loss during rollouts","Vendor-tuned package coverage may miss campus SKUs"],"security":"Constrain who can apply NodeWright CRs; verify package image signatures/provenance; limit blast radius with selectors.","proofOfValue":["Canary compartment succeeds with validation checks","Failed package surfaces node condition","Non-interruptible pods pause sequence as designed"]},"delivery":{"work":"Install chart, author packages/policies, label workloads, execute progressive rollout, archive status.","dependencies":["registry for packages","GitOps","monitoring of Jobs/conditions"],"ownership":"Platform/host-ops team owning node OS baseline.","skillsAdoption":"Train ops on NodeWright CRs, interruption budgets, and package validation failures.","governanceCheckpoints":["Change approval for interruptionBudget","Security review of package images","Rollback/stop criteria"],"acceptanceCriteria":["Canary meets batch threshold","No unlabeled critical job evictions","Failed validation stops compartment"],"risks":["Mis-labeled selectors drain wrong pool","Package bug requires node rebuild","PDB misconfiguration stalls fleets"]}},"retrievedAt":"2026-09-26T03:01:17Z","enrichedAt":"2026-09-26T03:10:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Safer host maintenance may reduce after-hours toil for platform staff; does not address end-user accessibility of AI apps.","procurementImplications":"Open-source Apache-2.0 tooling; still budget for ops skill and package validation—not a turnkey support SKU by itself.","operatingModelImplications":"Moves host change from spreadsheet runbooks to CRD/GitOps ownership beside GPU Operator.","sourceVerification":{"openedUrl":"https://developer.nvidia.com/blog/manage-kubernetes-node-fleets-with-nodewright/","referenceExcerpt":"NodeWright has run in production at NVIDIA as Skyhook and is available as open source","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}