NVIDIA Details Open-Source Attribution Tool for AI Workload Failures

NVIDIA has detailed Attribution, an open-source system that traces distributed AI workload failures to their initiating events, helping infrastructure teams distinguish causes from errors that spread across nodes.

The system begins by defining the workload attempt under review and collecting records tied to it. Inputs include application and system logs, GPU driver and Fabric Manager messages, scheduler history, and node and network metrics. Its structured diagnosis captures the outcome, root cause, affected nodes, symptoms and supporting evidence.

Attribution has two deployable parts. The NVIDIA Attribution Service processes one workload attempt at a time through a REST API. The NVIDIA Attribution Blueprint connects cluster schedulers with telemetry sources, gathers workload context and records the diagnosis. Deterministic rules make results reproducible and customizable. The system analyzes millions of workloads each week across hundreds of thousands of GPUs and is used across NVIDIA AI infrastructure.

In a multi-rank training example, a GPU reports an XID 149 hard fault before NCCL errors appear across other ranks. Attribution links the events, identifies XID 149 as the root cause and retains the NCCL failures as symptoms, while naming the responsible node.

A separate synthetic case includes three different NCCL errors in application logs and XID 149 error words in system logs from node-02. Those words decode to an NVLink signal integrity issue. An early filtering rule identifies the signal issue while leaving the NCCL events unchanged; final refinement surfaces the issue on node-02 and preserves the NCCL reports as supporting evidence.

Diagnosis quality depends on the completeness of the available telemetry. When multiple causes fit the evidence, the sub_domain field is ambiguous. When evidence is insufficient, the field is unattributed.

A separate NVIDIA Recovery service detects interruptions in distributed AI training. On one training cluster, its mean detection latency was 69 seconds.