NVIDIA Details Three-Stage Workflow for Kubernetes GPU Clusters

NVIDIA outlined a three-stage DSX OS workflow for setting up, checking and monitoring Kubernetes GPU clusters, giving operators ways to find configuration and node problems that utilization readings can miss. The company described the workflow on Nov. 5 as a way to address issues that can slow distributed AI training even when basic GPU health indicators look normal.

The system groups its functions into three Apache 2.0-licensed projects. NVIDIA AI Cluster Runtime (AICR) establishes a version-pinned baseline for cluster components such as networking, storage and drivers. It prepares deployment files for Helm and Argo CD, tools operators use to manage Kubernetes applications.

AICR recipes include dependency graphs and health assertions for components. The runtime’s baseline and drift checks help operators identify when a cluster configuration has changed from its intended setup. The guide builds on NVIDIA’s earlier AICR instructions for configuring Dynamo on an existing Kubernetes cluster, while adding readiness testing and node-health response to the workflow.

NVIDIA Cluster Readiness Engine (NVCRE) checks whether a cluster is ready for production workloads by running AI tasks and diagnostics. Those checks include NVIDIA Collective Communication Library (NCCL) tests, which can help identify faulty nodes. NVSentinel monitors cluster health during operations.

NVSentinel can automatically cordon or taint an unhealthy node, drain its workloads and trigger NVCRE revalidation before the node returns to service. The workflow connects cluster setup and readiness checks with ongoing monitoring and revalidation.

NVIDIA’s discussion also presents goodput, or completed useful work, as a way to assess efficiency beyond utilization. It does not report measured platform-wide performance gains or reductions in fleetwide failure rates.