arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.00611cs.DC
收藏

StarTrail:面向高效近无限上下文 Transformer 模型训练的同心环序列并行

StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training

National University of Singapore(新加坡国立大学);University of Southern California(南加州大学);University of California, Berkeley(加州大学伯克利分校)

Ziming Liu, Shaoyu Wang, Shenggan Cheng, Zhongkai Zhao, Kai Wang, Xuanlei Zhao, James Demmel, Yang You

AI总结:

针对长序列分布式 Transformer 训练的通信瓶颈,提出 StarTrail 同心环多维序列并行系统,在 NLP 与 CV 任务中显著提升训练性能。

摘要:

在分布式环境中基于长序列训练 Transformer 模型,在效率和可扩展性方面带来了重大挑战。现有方法要么受限于注意力头的数量,要么存在过高的通信开销。为解决这一问题,我们提出 StarTrail,这是一个面向长序列的多维同心分布式训练系统,可培育高效通信范式,并为通信安排提供额外的调优灵活性。具体而言,StarTrail 引入一个额外的并行维度,并将点对点通信划分为多个子环,从而大幅减少通信量并避免带宽瓶颈。通过在多种硬件环境以及自然语言处理(NLP)和计算机视觉(CV)任务上开展全面实验,我们证明该方法显著优于支持长序列长度的最先进方法:在不影响计算结果的情况下,在 GPT 风格模型上实现最高 77.12% 的性能提升,在 DiT(Diffusion Transformer)模型上实现最高 114.33% 的性能提升。

英文摘要:

Training Transformer models on long sequences in a distributed setting poses significant challenges in terms of efficiency and scalability. Current methods are either constrained by the number of attention heads or excessive communication overheads. To address this problem, we propose StarTrail, a multi-dimensional concentric distributed training system for long sequences, fostering an efficient communication paradigm and providing additional tuning flexibility for communication arrangements. Specifically, StarTrail introduces an extra parallel dimension and divides the peer-to-peer communication into sub-rings to substantially reduce communication volume and avoid bandwidth bottlenecks. Through comprehensive experiments across diverse hardware environments and on both Natural Language Processing (NLP) and Computer Vision (CV) tasks, we demonstrate that our approach significantly surpasses state-of-the-art methods that support Long sequence lengths, achieving performance improvements of up to 77.12% on GPT-style models and up to 114.33% on DiT (Diffusion Transformer) models without affecting the computations results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑