arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.06003cs.LGcs.AI
收藏

重新思考高效大语言模型训练中的内存与通信成本

Rethinking Memory and Communication Cost for Efficient Large Language Model Training

Chan Wu, Hanxiao Zhang, Lin Ju, Jinjing Huang, Youshao Xiao, Zhaoxin Huan, Siyuan Li, Fanzhuang Meng, Lei Liang, Xiaolu Zhang, Jun Zhou

AI总结:

本文提出内存-通信平衡策略集 PaRO 和分层重叠环通信拓扑 HO-Ring,通过细粒度分片与高效通信设计,在少量内存冗余下显著提升大语言模型分布式训练吞吐量和可扩展性。

摘要:

近年来,针对大语言模型训练的各种分布式策略相继被提出。然而,这些方法在内存消耗与通信成本之间的权衡方面提供的解决方案较为有限。本文重新思考了内存消耗和通信成本对大语言模型训练速度的影响,并提出了一种内存-通信平衡策略集——部分冗余优化器(PaRO)。PaRO 提供了全面的选项,通过细粒度分片策略,在引入少量内存冗余的情况下减少组间通信的数据量和频率,从而在多种训练场景下提升训练效率。此外,我们提出了一种分层重叠环(HO-Ring)通信拓扑,用于在大语言模型训练中提升节点之间或跨交换机之间的通信效率。实验表明,与当前最优(SOTA)方法相比,PaRO 将训练吞吐量显著提升了 1.19 倍至 2.50 倍,并实现了近线性的可扩展性。HO-Ring 算法相比传统 Ring 算法将通信效率提高了 36.5%。

英文摘要:

Recently, various distributed strategies for large language model training have been proposed. However, these methods provided limited solutions for the trade-off between memory consumption and communication cost. In this paper, we rethink the impact of memory consumption and communication costs on the training speed of large language models, and propose a memory-communication balanced strategy set Partial Redundancy Optimizer (PaRO). PaRO provides comprehensive options which reduces the amount and frequency of inter-group communication with minor memory redundancy by fine-grained sharding strategy, thereby improving the training efficiency in various training scenarios. Additionally, we propose a Hierarchical Overlapping Ring (HO-Ring) communication topology to enhance communication efficiency between nodes or across switches in large language model training. Our experiments demonstrate that PaRO significantly improves training throughput by 1.19x-2.50x compared to the SOTA method and achieves a near-linear scalability. The HO-Ring algorithm improves communication efficiency by 36.5% compared to the traditional Ring algorithm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑