{"resourceId":"nvidia-docs-nvsentinel-overview-monitoring-default-20261105","versions":[{"version":"external-a3cf20d052ba2889f9a21ce7e45d10baf12a483e43cd597498a6c416be0bca20","resource":{"id":"nvidia-docs-nvsentinel-overview-monitoring-default-20261105","title":"NVSentinel docs: default install is monitoring-only; quarantine, drain and remediation are opt-in as confidence grows","organization":"NVIDIA","sector":"GPU fault detection and remediation","geography":"Global (AWS, GCP, Azure, OCI and on-prem cited)","publishedAt":"2026-11-05","publicationDate":null,"eventDate":"2026-11-05","sourceName":"NVIDIA NVSentinel Documentation","sourceLabel":"NVIDIA Docs: NVSentinel Overview (v1.30.0)","sourceUrl":"https://docs.nvidia.com/nvsentinel/getting-started/overview","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["infrastructure","operating-model","data-security"],"finding":"Current NVSentinel documentation describes a Kubernetes GPU fault detection and remediation system that can cordon, drain and trigger break-fix. It claims deployments on AWS, GCP, Azure and OCI up to 1,100+ nodes and about 40,000 GPUs. Critically for operators, by default only health monitoring is enabled and is described as safe because it only observes and reports; fault quarantine, node drainer and fault remediation are enabled via Helm values as confidence grows. Newly relevant because the Nov 5 DSX OS post promotes enabling remediation after monitoring trust is established.","sledRelevance":"Gives campus and state platform teams vendor-stated defaults and scale claims to pressure-test before automated reboot of shared research nodes.","evidence":"Vendor product documentation; scale figures are NVIDIA-reported; living docs without a single publication date.","architectureImplications":"Modular pipeline (quarantine, drainer, remediation, janitor) coordinated through Kubernetes APIs; circuit breaker against mass quarantine.","governanceImplications":"Enabling remediation is an explicit policy choice; NVIDIA internal defaults (full remediation) differ from the public default.","securityPrivacyImplications":"Automated drain/reboot can disrupt tenants; false positives need an exit path (docs also cover ManagedByNVSentinel=false in related runbooks).","caveats":"Vendor docs; scale claims unverified independently; version Latest v1.30.0 at retrieval.","streamIds":["nvidia"],"roles":{"sales":"Lead with the default, not the maximum automation. NVSentinel documentation says monitoring-only is the safe default and remediation is opt-in. That framing fits risk-averse university and state shared clusters better than NVIDIA's internal full-remediation posture. Do not cite the 40,000 GPU figure as a customer outcome; treat it as a vendor scale claim to verify.","engineering":"Install in monitoring mode, export health events to existing alerting, and measure false positives before enabling quarantine or reboot. Review CEL rulesets and the circuit breaker. Separate NVIDIA's internal default (full remediation) from what a multi-tenant campus should enable on day one.","delivery":"Phase enablement with written approval gates. Start with observe-and-report, add quarantine with a circuit breaker, then remediation on a non-critical pool. Keep a runbook for false positives and nodes labeled out of management. Acceptance should include a measured false-positive rate and a tested rollback to monitoring-only."},"roleDetails":{"sales":{"customerProblem":"Shared GPU sites fear automated remediation more than they fear missing a bad GPU.","stakeholders":["SRE/platform lead","Research computing director","Security/compliance"],"discoveryQuestions":["Is automated node reboot allowed on tenant nodes?","What false-positive rate is acceptable?"],"valueHypothesis":"Monitoring-first adoption may surface faults without immediate job disruption.","potentialEngagement":"Remediation policy design and monitor-mode pilot.","unsupportedClaims":["Independent verification of 40,000 GPU scale","Zero false-positive quarantine","Guaranteed minutes-to-repair"]},"engineering":{"fit":"Kubernetes GPU fleets needing structured health events before automation.","architectureIntegration":"Helm chart with monitoring default; optional quarantine/drainer/remediation modules.","prerequisites":["DCGM and node log signals","On-call ownership","Circuit-breaker configuration"],"constraints":["Vendor scale claims","Remediation risk to tenants","Version-specific behavior"],"security":"Audit who can enable remediation Helm values; restrict break-fix actions.","proofOfValue":["Two-week monitor-mode false-positive review","Successful detection of a known injected fault without eviction"]},"delivery":{"work":"Monitor-mode rollout, then gated enablement of quarantine and remediation.","dependencies":["Alert routing","Change board approval for reboot automation"],"ownership":"Platform SRE owns NVSentinel; research computing approves tenant impact policy.","skillsAdoption":"Operators learn health-event triage and cordon runbooks.","governanceCheckpoints":["Approve leaving monitoring-only","Approve quarantine","Approve remediation/reboot"],"acceptanceCriteria":["Proposal: remediation Helm values remain false until false-positive rate is reviewed","Proposal: circuit breaker tested in a game-day"],"risks":["False-positive drains","Mass quarantine during fabric events","Configuration drift from NVIDIA internal defaults"]}},"retrievedAt":"2026-11-06T03:59:54Z","enrichedAt":"2026-11-06T04:05:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Requires SRE skills for Helm values, health-event triage and break-fix workflows.","procurementImplications":"Open component in DSX OS story; operational risk and on-call process matter more than license cost.","operatingModelImplications":"Supports a staged trust model: observe, then quarantine, then remediate.","sourceVerification":{"openedUrl":"https://docs.nvidia.com/nvsentinel/getting-started/overview","referenceExcerpt":"By default, only health monitoring is enabled. This is safe to deploy in any cluster as it only observes and reports.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}