arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.01614cs.LGcs.AI
收藏

在受限带宽网络中提升大语言模型训练的稳定性

Enhancing Stability for Large Language Models Training in Constrained Bandwidth Networks

Foundational AI Technologies (FAIT), LinkedIn(领英基础人工智能技术(FAIT));AI Platforms, LinkedIn(领英人工智能平台)

Yun Dai, Tejas Dharamsi, Byron Hsu, Tao Song, Hamed Firooz

AI总结:

针对低带宽集群中ZeRO++的hpZ方案因竞态条件导致大模型训练收敛不稳定的问题,提出分区算法改进方案,在保持训练效率的同时实现大模型可靠收敛,且不损失收敛质量。

摘要:

训练具有数十亿参数的超大型语言模型(LLM)是一项计算密集型任务,挑战着当前数据并行训练系统的极限。尽管ZeRO++等技术已能在低成本低带宽集群上高效分布式训练这类巨型模型,但由于用于减少跨机器通信的分层分区(hpZ)方案存在潜在竞态条件,这类技术可能遭遇收敛问题。在本研究中,我们首先揭示了在训练数十亿参数模型时,这些竞态条件如何导致不稳定。随后我们对分区算法提出改进,在保持有竞争力的训练效率的同时解决这些收敛难题。在训练数十亿参数的Falcon Models和Llama-2模型上的实证评估表明,改进后的算法能够在这些巨型模型上实现可靠收敛,而原版ZeRO++ hpZ无法收敛。该改进算法支持更大模型的稳健训练,可实现98%的吞吐量并提升模型训练速度,且不牺牲收敛质量。

英文摘要:

Training extremely large language models (LLMs) with billions of parameters is a computationally intensive task that pushes the limits of current data parallel training systems. While techniques like ZeRO++ have enabled efficient distributed training of such giant models on inexpensive low-bandwidth clusters, they can suffer from convergence issues due to potential race conditions in the hierarchical partitioning (hpZ) scheme employed to reduce cross-machine communication. In this work, we first show how these race conditions cause instability when training models with billions of parameters. We then propose a modification to the partitioning algorithm that addresses these convergence challenges while maintaining competitive training efficiency. Empirical evaluation on training the multi-billion parameters Falcon Models and Llama-2 models demonstrates the updated algorithm's ability to achieve reliable convergence on these massive models, where stock ZeRO++ hpZ fails to converge. The updated algorithm enables robust training of larger models with 98\% throughput and model training speed improvement without sacrificing the quality of convergence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑