NVIDIA Reshard Cuts DeepSeek-V3 Weight Refit to 2.6 Seconds

NVIDIA's Reshard collective reduced DeepSeek-V3 weight-refit time to 2.6 seconds on DGX H100 in the company's internal tests, addressing delays as reinforcement-learning systems pass updated models to inference GPUs. The conventional process took 14.9 seconds for the 671-billion-parameter model.

Weight refit moves updated model weights from the GPUs used for training to those used for inference. The inference system then uses the updated model to generate fresh training trajectories. This handoff is part of reinforcement-learning workflows, where models are trained using feedback from their outputs.

Traditional refit gathers complete model tensors before sending them to inference GPUs. That can move more data than the destination GPUs need when each uses only particular pieces, or shards, of a tensor. The communication takes place through GPU collective operations, in which participating GPUs coordinate data exchanges.

Reshard treats the handoff as a transfer between distributed tensor layouts. It maps the portions of data shared between the source and destination layouts, then sends those shards directly. This avoids building an intermediate version of the full tensor and is designed to reduce redundant communication while balancing transfers across GPUs.

NVIDIA's internal benchmarks also measured the 671-billion-parameter DeepSeek-V3 model on DGX GB200 systems. Conventional refit took 6.1 seconds there, compared with 2.3 seconds using Reshard. Across tests on DGX H100 and DGX GB200 systems, the reported speedups ranged from 2.7 to 5.7 times over conventional refit.

In a separate test on a 256-GPU NVL72 cluster with NDR InfiniBand, weight-synchronization time fell from 5.78 seconds to 2.77 seconds, while the full reinforcement-learning step was 12.7% shorter. A transfer for one feed-forward mixture-of-experts layer took 9.8 milliseconds, compared with 77.3 milliseconds using flat direct sends. NVIDIA has made an open-source implementation available through its NCCL extensions repository.