arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.07210cs.DCcs.AI
收藏

EDiT:一种基于 Local-SGD 的大语言模型高效分布式训练方法

EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models

Ant Group(蚂蚁集团)

Jialiang Cheng, Ning Gao, Yun Yue, Zhiling Ye, Jiadi Jiang, Jian Sha

AI总结:

本文提出 EDiT 及其异步变体 A-EDiT,通过定制 Local SGD、模型分片、前向传播逐层同步与伪梯度惩罚,降低通信和内存开销并稳定大规模 LLM 训练。

摘要:

分布式训练方法对大语言模型(LLM)至关重要。然而,现有的分布式训练方法往往受到通信瓶颈、掉队者以及有限弹性的困扰,尤其是在异构或大规模环境中。为解决这些问题,研究者提出了 Local SGD 方法,但其有效性仍局限于小规模训练,原因是存在额外的内存开销,且缺乏对效率和稳定性的关注。为应对这些问题,我们提出 EDiT,这是一种创新的高效分布式训练方法,将定制化 Local SGD 方法与模型分片技术相结合,以提升大规模训练效率。EDiT 在前向传播过程中执行逐层参数同步,从而降低通信和内存开销并实现重叠。此外,EDiT 采用伪梯度惩罚策略来抑制损失尖峰,这确保了训练稳定性并提升了性能。另外,我们提出 A-EDiT,它是 EDiT 的完全异步变体,可适配异构集群。基于 EDiT/A-EDiT,我们开展了一系列实验以验证 LLM 的大规模异步训练,并进行了全面分析。实验结果证明了 EDiT/A-EDiT 的优越性能,确立其作为多样化计算生态系统中分布式 LLM 训练的稳健解决方案。代码可在 Atorch 代码库获取:https://github.com/intelligent-machine-learning/atorch/tree/main/atorch/local_sgd。

英文摘要:

Distributed training methods are crucial for large language models (LLMs). However, existing distributed training methods often suffer from communication bottlenecks, stragglers, and limited elasticity, particularly in heterogeneous or large-scale environments. Local SGD methods have been proposed to address these issues, but their effectiveness remains limited to small-scale training due to additional memory overhead and lack of concerns on efficiency and stability. To tackle these issues, we propose EDiT, an innovative Efficient Distributed Training method that combines a tailored Local SGD approach with model sharding techniques to enhance large-scale training efficiency. EDiT performs layer-wise parameter synchronization during forward pass, reducing communication and memory overhead and enabling overlap. Besides, EDiT employs a pseudo gradient penalty strategy to suppress loss spikes, which ensures training stability and improves performance. Additionally, we introduce A-EDiT, a fully asynchronous variant of EDiT that accommodates heterogeneous clusters. Building on EDiT/A-EDiT, we conduct a series of experiments to validate large-scale asynchronous training for LLMs, accompanied by comprehensive analyses. Experimental results demonstrate the superior performance of EDiT/A-EDiT, establishing them as robust solutions for distributed LLM training in diverse computational ecosystems. The code is available at Atorch codebase: https://github.com/intelligent-machine-learning/atorch/tree/main/atorch/local_sgd.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑