NVIDIA Uses AI Agents to Trace Failures Across DGX SuperPOD Clusters

NVIDIA is using an AI operations framework inside DGX SuperPOD environments to examine training failures and infrastructure signals, helping site reliability engineers trace problems across the cluster. The design aims to reduce the manual effort involved in following a fault across separate infrastructure layers.

NVIDIA described its AI Factory Operations Agent on Wednesday, Nov. 4. The framework is organized around its NemoClaw blueprint, which coordinates agents and tools for distinct operational roles. They communicate through Model Context Protocol (MCP), a standard for connecting AI applications with external tools, and engineers can review results through a chat interface.

The Job Recovery Agent investigates failed training jobs, while a Hardware Agent checks for GPU faults, PCIe errors and thermal anomalies. An Observability Agent builds dashboards using Prometheus data and Grafana Loki logs. A Kubernetes agent can read cluster resources and scheduling status, while a Cluster Management Agent uses NVIDIA Base Command Manager APIs to diagnose and troubleshoot clusters.

The system presents possible causes and recommendations alongside evidence for engineers to assess. In production, the agents have detected failures and connected findings across workload, hardware and network layers. The framework keeps agent responsibilities distinct to limit overlap and makes evidence available for review.

The application runs in Kubernetes’ control plane. Kubernetes hosts its chat interface, OpenClaw gateway, MCP tool services, inference endpoint and NVIDIA OpenShell runtime; Helm handles deployment. OpenShell creates a separate Kubernetes sandbox for each shell-command session, limiting direct agent access to Kubernetes credentials.

Role-based access controls and server-side inspection tools further restrict operations. Agents can read pod and configmap information but cannot create pods or change existing ones. NVIDIA supports on-premises deployment and inference using locally served NVIDIA Nemotron open models. The modular framework can accommodate additional agents and infrastructure integrations without changes to existing components.