NVIDIA Mission Control 2.5 Adds Kubernetes Workload Recovery

NVIDIA’s Mission Control 2.5 extends automated recovery to Kubernetes workloads managed by NVIDIA Run:ai, giving AI computing clusters ways to respond to software failures and faulty hardware.

NVIDIA announced the update Oct. 26. The release brings the resiliency framework, previously used for jobs running on Slurm, into Kubernetes environments, where NVIDIA Run:ai serves as the default scheduler.

The service weighs job characteristics, prior failures and placement relationships before choosing whether to restart a job, retry it, return it to the queue or place it elsewhere.

A separate hardware-recovery service monitors compute nodes and GPUs, removes suspected faulty equipment from scheduling, and runs diagnostics or remediation. If hardware fails during training, the system can restart or reschedule the job on healthy nodes while isolating the affected node. For temporary software faults, it can retry without isolating healthy hardware. It can also take a degrading node out of scheduling before another job is assigned to it.

The release adds recovery support for the KAI scheduler with Kubeflow PyTorch training operators. It includes Run:ai 2.26 and Base Command Manager 11.34.0, and supports asynchronous checkpointing in NVRx 0.6.0. NVRx is installed separately and is not bundled with Mission Control.

Support for Vera Rubin NVL72 is in early access and limited to essential Base Command Manager bring-up. Run:ai and the Autonomous Resiliency Engine are not supported on that platform in this release. Minerva network telemetry and AIOps, along with NVIDIA Dynamic Power Software, are pre-release betas that are not included in the release and are available only through a scoped engagement.