arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.10197cs.DCcs.AI
收藏

揭秘分布式Transformer模型的通信特性

Demystifying the Communication Characteristics for Distributed Transformer Models

The Ohio State University(俄亥俄州立大学)

Quentin Anthony, Benjamin Michalowicz, Jacob Hatef, Lang Xu, Mustafa Abduljabbar, Aamir Shafi, Hari Subramoni, Dhabaleswar Panda

AI总结:

本文针对分布式Transformer训练的通信瓶颈,以GPT类语言模型为案例,研究不同并行方案的通信行为,用解析模型验证实证结果,揭示多因素相关性并指明优化方向。

摘要:

基于Transformer架构的深度学习(DL)模型已革新了大语言模型(LLMs)、视觉Transformer、音频生成和时间序列预测等众多深度学习应用。分布式训练是推动这一进展的重要动力,但分布式通信仍是训练进程的主要瓶颈。本文研究Transformer模型的通信行为,即多节点/多GPU深度学习训练中采用的不同并行方案在Transformer场景下如何传输数据。鉴于基于GPT的语言模型的普遍性,我们将其作为Transformer架构的案例研究对象。我们使用解析模型验证了从通信日志中得到的实证结果。总体而言,分析表明需进一步优化小消息点对点通信,揭示了序列长度、单GPU吞吐量、模型规模与所用优化方法之间的相关性,并指明了框架及高性能计算(HPC)中间件设计与优化中可进一步开展优化的方向。

英文摘要:

Deep learning (DL) models based on the transformer architecture have revolutionized many DL applications such as large language models (LLMs), vision transformers, audio generation, and time series prediction. Much of this progress has been fueled by distributed training, yet distributed communication remains a substantial bottleneck to training progress. This paper examines the communication behavior of transformer models - that is, how different parallelism schemes used in multi-node/multi-GPU DL Training communicate data in the context of transformers. We use GPT-based language models as a case study of the transformer architecture due to their ubiquity. We validate the empirical results obtained from our communication logs using analytical models. At a high level, our analysis reveals a need to optimize small message point-to-point communication further, correlations between sequence length, per-GPU throughput, model size, and optimizations used, and where to potentially guide further optimizations in framework and HPC middleware design and optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑