PyTorch 2.15 Adds Flight Recorder for GPU Training Hangs

PyTorch 2.15, released Oct. 28, adds automatic post-mortem diagnostic dumps for stalled distributed GPU training jobs. When a collective operation hangs or times out, Flight Recorder automatically saves diagnostic state for engineers to examine after the stall.

Flight Recorder is a 2.15 addition that works across communications backends. Version 2.15 adds a hang-triggered path for collecting state.

Distributed training lets multiple GPUs coordinate while working on the same model. Collective operations are exchanges used to keep participating GPUs in sync, and every GPU in a collective must take part. A stall in one of these operations can leave the coordinated job waiting. On NVIDIA hardware, the NVIDIA Collective Communications Library (NCCL) handles those communications.

PyTorch 2.15 also makes its redesigned NCCL backend the default for NCCL process groups. The backend was optional in PyTorch 2.14, so teams had to select it to use it. With the new version, groups using NCCL follow the redesigned path by default. This change applies to NCCL process groups, while Flight Recorder can work across communications backends.

Teams that need the former backend behavior can use fallback options in version 2.15. Those options are scheduled for removal in the next release.

The update also adds faster ways to reconfigure a training cluster after a failure. These features require NCCL 2.29 or later, or RCCL version 2.30.7 on ROCm.