NVIDIA Recovery Averaged 69 Seconds to Detect Training Anomalies on One Cluster

NVIDIA introduced NVIDIA Recovery on Monday, Nov. 2, as an open-source service in its DSX OS portfolio for detecting interruptions in distributed AI training. On one cluster, anomaly detection averaged 69 seconds.

The service combines application logs, GPU metrics and scheduler signals to identify crashes, hangs and stragglers, then coordinate a recovery response across the cluster. Recovery is licensed under Apache 2.0, but remains experimental and is not ready for production use. Documented support covers Slurm and Slurm on Kubernetes through Slinky; native Kubernetes workload recovery has not been implemented.

During monitoring from July 9 through Oct. 9, the cluster ran nearly 200 jobs per day and recorded about 8,550 anomaly detections. Mean detection latency was 69 seconds, while the average time to an automated recovery action was 94 seconds. The cluster’s size and workload mix were not disclosed.

The 94-second measure ends when the automated action occurs, rather than when a job returns to productive training. Scheduling and checkpoint restoration can add time before work resumes. Recovery does not create or restore checkpoints; the application handles checkpoint loading. No recovery success rate or time to resumed training was disclosed.

A separate 11-month analysis covered clusters with 16,000 and 8,000 GPUs, across 4 million jobs and more than 150 million A100 GPU-hours. Hardware-attributed failures affected fewer than 1% of jobs but accounted for 19% of GPU runtime. The analysis did not evaluate NVIDIA Recovery.

An illustrative calculation used a different measure: assuming each GPU had a 99.9% chance of operating without failure on a given day, a 10,000-GPU job had about a 0.005% chance of finishing the day without interruption. This was a hypothetical example, not a measured result for Recovery.

The new project has a different scope from NVIDIA’s Mission Control 2.5 recovery engine for Run:ai-managed Kubernetes workloads, covered in the Mission Control 2.5 update for Run:ai-managed Kubernetes workloads.