arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00277cs.DCcs.AI
收藏

在100,000块GPU上使用容错混合共享数据并行训练LLMs

Training LLMs with Fault Tolerant HSDP on 100,000 GPUs

Omkar Salpekar, Rohan Varma, Kenny Yu, Vladimir Ivanov, Yang Wang, Ahmed Sharif, Min Si, Shawn Xu, Feng Tian, Shengbao Zheng, Tristan Rice, Ankush Garg, Shangfu… 展开作者

Omkar Salpekar, Rohan Varma, Kenny Yu, Vladimir Ivanov, Yang Wang, Ahmed Sharif, Min Si, Shawn Xu, Feng Tian, Shengbao Zheng, Tristan Rice, Ankush Garg, Shangfu Peng, Shreyas Siravara, Wenyin Fu, Rodrigo de Castro, Adithya Gangidi, Andrey Obraztsov, Sharan Narang, Sergey Edunov, Maxim Naumov, Chunqiang Tang, Mathew Oldham

AI总结:

在100,000块GPU上提出容错混合共享数据并行训练方法,通过减少故障恢复停滞时间提升训练效率

摘要:

大规模训练系统通常使用同步训练,要求所有GPU同时正常工作。在我们的经验中,使用O(100K)块GPU进行同步训练会导致效率低下,因为频繁的故障和较长的恢复时间。为了解决这个问题,我们提出了一种新的训练范式,即容错混合共享数据并行(FT-HSDP)。FT-HSDP将数据并行副本作为容错单元。当发生故障时,只有包含故障GPU或服务器的单个数据并行副本被停机并重新启动,而其他副本继续训练。为了在大规模上实现这一想法,FT-HSDP结合了几种技术:1)我们引入了容错All Reduce(FTAR)协议,用于在数据并行副本之间交换梯度。FTAR依靠CPU驱动复杂的控制逻辑,用于添加或移除参与者动态,同时依赖GPU进行数据传输以获得最佳性能。2)我们引入了非阻塞追赶协议,允许恢复副本以最小的停滞时间加入训练。与完全同步训练相比,FT-HSDP在O(100K)块GPU上将由于故障恢复导致的停滞时间从10分钟减少到3分钟,有效训练时间从44%增加到80%。我们进一步证明,FT-HSDP的异步恢复不会对结果模型的准确性造成任何意义的降级。

英文摘要:

Large-scale training systems typically use synchronous training, requiring all GPUs to be healthy simultaneously. In our experience training on O(100K) GPUs, synchronous training results in a low efficiency due to frequent failures and long recovery time. To address this problem, we propose a novel training paradigm, Fault Tolerant Hybrid-Shared Data Parallelism (FT-HSDP). FT-HSDP uses data parallel replicas as units of fault tolerance. When failures occur, only a single data-parallel replica containing the failed GPU or server is taken offline and restarted, while the other replicas continue training. To realize this idea at scale, FT-HSDP incorporates several techniques: 1) We introduce a Fault Tolerant All Reduce (FTAR) protocol for gradient exchange across data parallel replicas. FTAR relies on the CPU to drive the complex control logic for tasks like adding or removing participants dynamically, and relies on GPU to perform data transfer for best performance. 2) We introduce a non-blocking catch-up protocol, allowing a recovering replica to join training with minimal stall. Compared with fully synchronous training at O(100K) GPUs, FT-HSDP can reduce the stall time due to failure recovery from 10 minutes to 3 minutes, increasing effective training time from 44\% to 80\%. We further demonstrate that FT-HSDP's asynchronous recovery does not bring any meaning degradation to the accuracy of the result model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑