arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.03488cs.DC
收藏

Seq1F1B:面向大语言模型训练的高效序列级流水线并行

Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training

Ao Sun, Weilin Zhao, Xu Han, Cheng Yang, Xinrong Zhang, Zhiyuan Liu, Chuan Shi, Maosong Sun

AI总结:

针对长序列LLM训练中流水线并行的内存与气泡瓶颈,提出序列级1F1B调度方法Seq1F1B,通过细粒度切分和计算感知划分,在64块A100上以更低内存训练30B模型至64k序列,吞吐量优于Megatron 1F1B。

摘要:

大语言模型(LLM)的出现高度依赖于分布式训练策略,其中流水线并行发挥着至关重要的作用。随着LLM训练序列长度扩展到32k甚至128k,当前的流水线并行方法面临严重瓶颈,包括高内存占用和大量流水线气泡,极大地阻碍了模型的可扩展性和训练吞吐量。为了提升内存效率和训练吞吐量,在本工作中,我们提出了一种针对长序列LLM训练的高效序列级一前向一后向(1F1B)流水线调度方法,命名为Seq1F1B。Seq1F1B将批级可调度单元分解为更细粒度的序列级单元,从而减小气泡大小和内存占用。考虑到若序列均匀切分,Seq1F1B可能产生轻微额外气泡,我们设计了一种计算感知策略来划分输入序列以缓解这一副作用。与Megatron 1F1B流水线并行等竞争性流水线基线方法相比,我们的方法以更少的内存占用实现了更高的训练吞吐量。值得注意的是,Seq1F1B能够在64块NVIDIA A100 GPU上高效训练具有300亿参数的LLM,序列长度可达64k,且无需重计算策略,这是现有方法无法实现的壮举。我们的源代码基于Megatron-LM,现可在https://github.com/MayDomine/Seq1F1B.git获取。

英文摘要:

The emergence of large language models (LLMs) relies heavily on distributed training strategies, among which pipeline parallelism plays a crucial role. As LLMs' training sequence length extends to 32k or even 128k, the current pipeline parallel methods face severe bottlenecks, including high memory footprints and substantial pipeline bubbles, greatly hindering model scalability and training throughput. To enhance memory efficiency and training throughput, in this work, we introduce an efficient sequence-level one-forward-one-backward (1F1B) pipeline scheduling method tailored for training LLMs on long sequences named Seq1F1B. Seq1F1B decomposes batch-level schedulable units into finer sequence-level units, reducing bubble size and memory footprint. Considering that Seq1F1B may produce slight extra bubbles if sequences are split evenly, we design a computation-wise strategy to partition input sequences and mitigate this side effect. Compared to competitive pipeline baseline methods such as Megatron 1F1B pipeline parallelism, our method achieves higher training throughput with less memory footprint. Notably, Seq1F1B efficiently trains a LLM with 30B parameters on sequences up to 64k using 64 NVIDIA A100 GPUs without recomputation strategies, a feat unachievable with existing methods. Our source code is based on Megatron-LM, and now is avaiable at: https://github.com/MayDomine/Seq1F1B.git.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑