From the NVIDIA edition of November 5, 2026
NVSentinel docs: default install is monitoring-only; quarantine, drain and remediation are opt-in as confidence grows
NVIDIA · GPU fault detection and remediation · Global (AWS, GCP, Azure, OCI and on-prem cited)
- Publisher
- NVIDIA NVSentinel Documentation
- Original publication
- 2026-11-05
- Source retrieved
- 2026-11-06
- Event date
- 2026-11-05
What happened
Current NVSentinel documentation describes a Kubernetes GPU fault detection and remediation system that can cordon, drain and trigger break-fix. It claims deployments on AWS, GCP, Azure and OCI up to 1,100+ nodes and about 40,000 GPUs. Critically for operators, by default only health monitoring is enabled and is described as safe because it only observes and reports; fault quarantine, node drainer and fault remediation are enabled via Helm values as confidence grows. Newly relevant because the Nov 5 DSX OS post promotes enabling remediation after monitoring trust is established.
Why it matters
Gives campus and state platform teams vendor-stated defaults and scale claims to pressure-test before automated reboot of shared research nodes.
Evidence and measured results
Vendor product documentation; scale figures are NVIDIA-reported; living docs without a single publication date.
Limitations and uncertainty
Vendor docs; scale claims unverified independently; version Latest v1.30.0 at retrieval.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-06; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Shared GPU sites fear automated remediation more than they fear missing a bad GPU.
- Stakeholders
- SRE/platform lead
- Research computing director
- Security/compliance
- Discovery questions
- Is automated node reboot allowed on tenant nodes?
- What false-positive rate is acceptable?
- Value hypothesis
- Monitoring-first adoption may surface faults without immediate job disruption.
- Potential engagement
- Remediation policy design and monitor-mode pilot.
- Unsupported claims
- Independent verification of 40,000 GPU scale
- Zero false-positive quarantine
- Guaranteed minutes-to-repair
Pre-sales engineering
Role takeaway
- Fit
- Kubernetes GPU fleets needing structured health events before automation.
- Architecture and integration
- Helm chart with monitoring default; optional quarantine/drainer/remediation modules.
- Prerequisites
- DCGM and node log signals
- On-call ownership
- Circuit-breaker configuration
- Constraints
- Vendor scale claims
- Remediation risk to tenants
- Version-specific behavior
- Security
- Audit who can enable remediation Helm values; restrict break-fix actions.
- Proof of value
- Two-week monitor-mode false-positive review
- Successful detection of a known injected fault without eviction
Delivery
Role takeaway
- Work
- Monitor-mode rollout, then gated enablement of quarantine and remediation.
- Dependencies
- Alert routing
- Change board approval for reboot automation
- Ownership
- Platform SRE owns NVSentinel; research computing approves tenant impact policy.
- Skills and adoption
- Operators learn health-event triage and cordon runbooks.
- Governance checkpoints
- Approve leaving monitoring-only
- Approve quarantine
- Approve remediation/reboot
- Acceptance criteria
- Proposal: remediation Helm values remain false until false-positive rate is reviewed
- Proposal: circuit breaker tested in a game-day
- Risks
- False-positive drains
- Mass quarantine during fabric events
- Configuration drift from NVIDIA internal defaults
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Modular pipeline (quarantine, drainer, remediation, janitor) coordinated through Kubernetes APIs; circuit breaker against mass quarantine.
Governance
Who approves, reviews and stays accountable for outcomes?
Enabling remediation is an explicit policy choice; NVIDIA internal defaults (full remediation) differ from the public default.
Security and privacy
What data, permissions and controls need testing?
Automated drain/reboot can disrupt tenants; false positives need an exit path (docs also cover ManagedByNVSentinel=false in related runbooks).
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Requires SRE skills for Helm values, health-event triage and break-fix workflows.
Procurement
What should contracts, pricing and exit terms secure?
Open component in DSX OS story; operational risk and on-call process matter more than license cost.
Operating model
Which teams own the service once it runs?
Supports a staged trust model: observe, then quarantine, then remediate.
Publication history
- 2026-11-05NVIDIA · Issue 616 resources
Stable resource ID: nvidia-docs-nvsentinel-overview-monitoring-default-20261105