{"resourceId":"github-nvidia-cluster-readiness-engine-nvcre-non-goals-20261105","versions":[{"version":"external-a3cf20d052ba2889f9a21ce7e45d10baf12a483e43cd597498a6c416be0bca20","resource":{"id":"github-nvidia-cluster-readiness-engine-nvcre-non-goals-20261105","title":"NVCRE GitHub: workload certification reports failing nodes but never cordons; not continuous monitoring or a benchmark leaderboard","organization":"NVIDIA","sector":"Open-source GPU cluster certification","geography":"Global","publishedAt":"2026-11-05","publicationDate":null,"eventDate":"2026-11-05","sourceName":"GitHub","sourceLabel":"GitHub: NVIDIA/cluster-readiness-engine README","sourceUrl":"https://github.com/NVIDIA/cluster-readiness-engine","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["infrastructure","operating-model"],"finding":"The NVIDIA Cluster Readiness Engine repository states NVCRE runs real training and NCCL workloads across topology-aware node groups, reports every bad node with a reason, and leaves quarantine to the platform: it never cordons, taints or otherwise modifies a node. Scope and non-goals say it is not continuous monitoring, not a general training platform, and not a benchmark leaderboard—measurements exist to find faults, not to rank hardware. Newly relevant as the companion limit statement to NVIDIA's Nov 5 DSX OS configure-certify-operate post.","sledRelevance":"Clarifies what campus and state platform teams should and should not expect from a certification gate beside NVSentinel or local remediation tooling.","evidence":"Vendor open-source README inspected Nov 5; living document without a fixed publication date.","architectureImplications":"Use NVCRE as a one-shot or on-demand certification pass; pair with separate continuous monitoring and remediation.","governanceImplications":"Sites must decide who acts on failed-node reports; NVCRE will not quarantine for them.","securityPrivacyImplications":"Burn-in workloads consume GPU time and may process model artifacts; isolate certification namespaces.","caveats":"Vendor documentation; no independent measurement of detection quality; living README may change.","streamIds":["nvidia"],"roles":{"sales":"Use the README's non-goals to set expectations. NVCRE finds faulty nodes under load and reports reasons; it does not monitor continuously or quarantine nodes. For research computing buyers evaluating the Nov 5 DSX OS story, this is the boundary document: certification is not remediation. Avoid selling it as a complete ops platform.","engineering":"Integrate NVCRE as a certification controller that drives TrainJobs/NCCL tests and writes results without mutating nodes. Confirm GPU Operator labels for architecture detection. Plan who consumes failure reports—NVSentinel, a custom controller, or human runbooks—because quarantine is explicitly out of scope.","delivery":"Stand up a certification catalog on a non-production pool first. Define who reviews failed-node reasons and what threshold constitutes a gate failure. Do not promise leaderboard-style rankings from NVCRE measurements; keep acceptance criteria tied to fault finding and tenant-handover readiness."},"roleDetails":{"sales":{"customerProblem":"Buyers conflate cluster certification tools with continuous remediation platforms.","stakeholders":["Platform engineering lead","Research computing director"],"discoveryQuestions":["Who cordons nodes today when burn-in fails?","Do you need continuous monitoring or pre-handover certification?"],"valueHypothesis":"Clear non-goals reduce mis-scoped purchases and failed pilots.","potentialEngagement":"Scope workshop separating certification from remediation tooling.","unsupportedClaims":["Continuous monitoring","Automatic quarantine","Hardware ranking benchmarks"]},"engineering":{"fit":"Pre-production and post-repair certification for GPU Kubernetes pools.","architectureIntegration":"Kubernetes controller plus nvcrectl; no node mutation; platform handles quarantine.","prerequisites":["GPU Operator","Kubeflow Trainer dependency per README","Topology-aware node grouping"],"constraints":["Not continuous monitoring","Consumes GPU time during tests","Living vendor docs"],"security":"Isolate certification workloads; restrict who can create Certification resources.","proofOfValue":["Failed-node reasons match subsequent physical faults","No unexpected cordons from NVCRE itself"]},"delivery":{"work":"Document the certify-then-remediate handoff and pilot on one rack or tenant pool.","dependencies":["Platform remediation owner","Test image registry access"],"ownership":"Platform team owns NVCRE; ops/SRE owns quarantine actions.","skillsAdoption":"Operators learn to read Certification reports rather than manual NCCL bisection.","governanceCheckpoints":["Approve which catalogs run in production windows","Approve who may ignore a failed node"],"acceptanceCriteria":["Proposal: NVCRE never mutates nodes in the pilot namespace","Proposal: every failed node has a recorded owner response within 1 business day"],"risks":["Burn-in starving research queues","Reports ignored without quarantine automation"]}},"retrievedAt":"2026-11-06T03:59:54Z","enrichedAt":"2026-11-06T04:05:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Requires Kubernetes and GPU ops skills to interpret failure reasons and CEL-related platform actions.","procurementImplications":"Apache-2.0 project; operational cost is GPU time for burn-in and staff to act on reports.","operatingModelImplications":"Encourages a clear handoff: certify, then separately remediate.","sourceVerification":{"openedUrl":"https://github.com/NVIDIA/cluster-readiness-engine","referenceExcerpt":"Quarantine is left to your platform: NVCRE never cordons, taints, or otherwise modifies a node.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}