{"resourceId":"nvidia-attribution-open-source-workload-failure-diagnosis-20261102","versions":[{"version":"external-e1c4c959232b1a856a746defcf560b286b9241c159f8688e8ba5e3e08793e2e3","resource":{"id":"nvidia-attribution-open-source-workload-failure-diagnosis-20261102","title":"NVIDIA open-sources Attribution, its rules-based tool for finding the root cause of distributed AI job failures, and says results depend on telemetry quality","organization":"NVIDIA","sector":"AI infrastructure operations software","geography":"Global","publishedAt":"2026-11-02","publicationDate":"2026-11-02","eventDate":"2026-11-02","sourceName":"NVIDIA Technical Blog","sourceLabel":"NVIDIA blog: Diagnose Distributed AI Workload Failures with NVIDIA Attribution","sourceUrl":"https://developer.nvidia.com/blog/diagnose-distributed-ai-workload-failures-with-nvidia-attribution/","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["infrastructure","operating-model"],"finding":"NVIDIA open-sourced NVIDIA Attribution, which it says it has used in production for years across hundreds of thousands of GPUs, analysing millions of workloads each week. It correlates workload-scoped application logs, syslogs, GPU driver and Fabric Manager messages, scheduler history and metrics, and uses deterministic rules to separate the root cause from downstream symptoms. For example, it identifies an XID 149 NVLink signal-integrity fault on one node as the cause of NCCL errors on other ranks. It ships as a self-hosted REST service plus a Python reference Blueprint with connectors for Slurm, Kueue, Loki, local files, Kubernetes pod logs and PromQL, and output to stdout or SQLite. When several causes fit the evidence, the result is marked ambiguous, and NVIDIA says attribution quality depends on telemetry completeness.","sledRelevance":"Useful to university and state HPC teams that spend staff time working out whether a failed GPU job was hardware, system software or user code. The deterministic, inspectable rules suit public operators who must explain incidents. Limited relevance outside sites running multi-node GPU work.","evidence":"Vendor blog with a synthetic worked example and an unquantified production anecdote (fabric port patterns in one RDMA cluster). No accuracy, precision or time-to-diagnosis figures.","architectureImplications":"Blueprint runs near cluster data sources and calls a shared or per-cluster Attribution Service; outputs structured terminal_state and sub_domain fields for downstream systems.","governanceImplications":"Deterministic rules and an evidence trail make diagnoses reviewable; ambiguous results are labelled rather than forced.","securityPrivacyImplications":"Collects application logs and syslogs, which can contain user data or paths; collection scope and retention need setting.","caveats":"No published accuracy metrics; rules are tuned to NVIDIA hardware failure modes (e.g., GB200 priority rules); the worked example uses synthetic data.","streamIds":["nvidia"],"roles":{"sales":"Frame this as help for small HPC teams that cannot afford a dedicated reliability engineer. NVIDIA Attribution is free and explains why distributed jobs failed using rules staff can read. NVIDIA reports long production use but gives no accuracy figures, so the pitch is a diagnostic pilot, not a promise of faster fixes. A bounded engagement could connect Attribution to one cluster's logs and compare its verdicts with recent incident tickets. Avoid claiming it works equally well on mixed-vendor hardware; the built-in rules target NVIDIA failure modes.","engineering":"Assess telemetry first, because NVIDIA states attribution quality depends on it. The Blueprint needs scheduler context from Slurm or Kueue and logs from Loki, files or Kubernetes pods, plus PromQL metrics; outputs go to stdout or SQLite. Deploy the Service per cluster or shared. Proof of value: replay a set of past failures with known causes and measure how many verdicts match, how many come back ambiguous, and how many are wrong. Check that GPU driver and Fabric Manager logs are actually collected. Limit log collection scope for privacy.","delivery":"Run this as a back-testing exercise before trusting live verdicts. Build a labelled set of past incidents, replay them, and agree an accuracy bar with operations. The HPC operations lead owns rules and connector configuration. Staff need training to read verdicts and write rules for local failure modes. Feed results into the incident process, and pair with NVIDIA Recovery only after both are validated. Risks are telemetry gaps producing confident but wrong diagnoses, and rule drift as hardware generations change."},"roleDetails":{"sales":{"customerProblem":"HPC teams spend scarce staff time triaging cascades of GPU job errors to find a root cause.","stakeholders":["HPC operations lead","Research computing director","Hardware vendor support contacts"],"discoveryQuestions":["How long does root-causing a failed multi-node job take today?","Which logs (driver, fabric, scheduler) do you keep?","How do you track repeat hardware faults?"],"valueHypothesis":"Readable, rules-based diagnoses could shorten triage for common hardware faults.","potentialEngagement":"Back-test Attribution against recent incident tickets on one cluster.","unsupportedClaims":["Diagnostic accuracy","Time savings","Equal coverage for non-NVIDIA hardware"]},"engineering":{"fit":"Fits clusters with centralised logs and Slurm or Kueue scheduling.","architectureIntegration":"Python Blueprint collects context and observations; REST Attribution Service returns verdicts; SQLite or stdout outputs.","prerequisites":["Scheduler history access","Centralised logs including GPU driver and Fabric Manager messages","PromQL metrics"],"constraints":["No published accuracy","Rules tuned to NVIDIA failure modes","Reference implementation needing customisation"],"security":"Log collection may capture user data; restrict scope and retention.","proofOfValue":["Replay labelled past failures and score match, ambiguous and wrong verdicts","Confirm required log sources are present before scoring"]},"delivery":{"work":"Back-test on labelled incidents, then use verdicts in live incident triage.","dependencies":["Telemetry pipeline","Incident history","Operations staff time"],"ownership":"HPC operations lead.","skillsAdoption":"Training on reading verdicts and writing local rules.","governanceCheckpoints":["Accuracy review against labelled incidents","Privacy review of collected logs"],"acceptanceCriteria":["Proposed: verdicts match known root cause in at least 80% of replayed incidents","Proposed: ambiguous verdicts reviewed within one business day"],"risks":["Confident wrong verdicts from incomplete telemetry","Rule drift across hardware generations"]}},"retrievedAt":"2026-11-03T04:00:35Z","enrichedAt":"2026-11-03T04:04:41Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Encodes expert failure knowledge into rules, helping smaller teams, but rule authoring needs skilled staff.","procurementImplications":"Open source; integration and rule maintenance are the real costs.","operatingModelImplications":"Gives operations teams a common diagnosis record to feed incident, repair and recovery systems.","sourceVerification":{"openedUrl":"https://developer.nvidia.com/blog/diagnose-distributed-ai-workload-failures-with-nvidia-attribution/","referenceExcerpt":"Quality of attribution is dependent on the quality and completeness of the underlying telemetry.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}