From the NVIDIA edition of November 2, 2026
NVIDIA open-sources Attribution, its rules-based tool for finding the root cause of distributed AI job failures, and says results depend on telemetry quality
NVIDIA · AI infrastructure operations software · Global
- Publisher
- NVIDIA Technical Blog
- Original publication
- 2026-11-02
- Source retrieved
- 2026-11-03
- Event date
- 2026-11-02
What happened
NVIDIA open-sourced NVIDIA Attribution, which it says it has used in production for years across hundreds of thousands of GPUs, analysing millions of workloads each week. It correlates workload-scoped application logs, syslogs, GPU driver and Fabric Manager messages, scheduler history and metrics, and uses deterministic rules to separate the root cause from downstream symptoms. For example, it identifies an XID 149 NVLink signal-integrity fault on one node as the cause of NCCL errors on other ranks. It ships as a self-hosted REST service plus a Python reference Blueprint with connectors for Slurm, Kueue, Loki, local files, Kubernetes pod logs and PromQL, and output to stdout or SQLite. When several causes fit the evidence, the result is marked ambiguous, and NVIDIA says attribution quality depends on telemetry completeness.
Why it matters
Useful to university and state HPC teams that spend staff time working out whether a failed GPU job was hardware, system software or user code. The deterministic, inspectable rules suit public operators who must explain incidents. Limited relevance outside sites running multi-node GPU work.
Evidence and measured results
Vendor blog with a synthetic worked example and an unquantified production anecdote (fabric port patterns in one RDMA cluster). No accuracy, precision or time-to-diagnosis figures.
Limitations and uncertainty
No published accuracy metrics; rules are tuned to NVIDIA hardware failure modes (e.g., GB200 priority rules); the worked example uses synthetic data.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-03; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- HPC teams spend scarce staff time triaging cascades of GPU job errors to find a root cause.
- Stakeholders
- HPC operations lead
- Research computing director
- Hardware vendor support contacts
- Discovery questions
- How long does root-causing a failed multi-node job take today?
- Which logs (driver, fabric, scheduler) do you keep?
- How do you track repeat hardware faults?
- Value hypothesis
- Readable, rules-based diagnoses could shorten triage for common hardware faults.
- Potential engagement
- Back-test Attribution against recent incident tickets on one cluster.
- Unsupported claims
- Diagnostic accuracy
- Time savings
- Equal coverage for non-NVIDIA hardware
Pre-sales engineering
Role takeaway
- Fit
- Fits clusters with centralised logs and Slurm or Kueue scheduling.
- Architecture and integration
- Python Blueprint collects context and observations; REST Attribution Service returns verdicts; SQLite or stdout outputs.
- Prerequisites
- Scheduler history access
- Centralised logs including GPU driver and Fabric Manager messages
- PromQL metrics
- Constraints
- No published accuracy
- Rules tuned to NVIDIA failure modes
- Reference implementation needing customisation
- Security
- Log collection may capture user data; restrict scope and retention.
- Proof of value
- Replay labelled past failures and score match, ambiguous and wrong verdicts
- Confirm required log sources are present before scoring
Delivery
Role takeaway
- Work
- Back-test on labelled incidents, then use verdicts in live incident triage.
- Dependencies
- Telemetry pipeline
- Incident history
- Operations staff time
- Ownership
- HPC operations lead.
- Skills and adoption
- Training on reading verdicts and writing local rules.
- Governance checkpoints
- Accuracy review against labelled incidents
- Privacy review of collected logs
- Acceptance criteria
- Proposed: verdicts match known root cause in at least 80% of replayed incidents
- Proposed: ambiguous verdicts reviewed within one business day
- Risks
- Confident wrong verdicts from incomplete telemetry
- Rule drift across hardware generations
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Blueprint runs near cluster data sources and calls a shared or per-cluster Attribution Service; outputs structured terminal_state and sub_domain fields for downstream systems.
Governance
Who approves, reviews and stays accountable for outcomes?
Deterministic rules and an evidence trail make diagnoses reviewable; ambiguous results are labelled rather than forced.
Security and privacy
What data, permissions and controls need testing?
Collects application logs and syslogs, which can contain user data or paths; collection scope and retention need setting.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Encodes expert failure knowledge into rules, helping smaller teams, but rule authoring needs skilled staff.
Procurement
What should contracts, pricing and exit terms secure?
Open source; integration and rule maintenance are the real costs.
Operating model
Which teams own the service once it runs?
Gives operations teams a common diagnosis record to feed incident, repair and recovery systems.
Publication history
- 2026-11-02NVIDIA · Issue 585 resources
Stable resource ID: nvidia-attribution-open-source-workload-failure-diagnosis-20261102