SkipPipe:异构网络中训练大语言模型的部分流水线与重排序流水线框架
SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks
AI总结:
针对异构网络中的LLM训练,提出SkipPipe部分流水线框架,通过阶段跳过与路径调度减少训练时间,保持收敛,实验显示迭代时间最多降低55%。
摘要:
数据和流水线并行在分布式节点上训练大语言模型(LLM)中无处不在。受成本效益训练需求的驱动,近期工作探索了端到端训练的高效通信安排。受LLM对层跳过和层重排序的抵抗性的启发,本文探讨了流水线训练中的阶段(若干连续层)跳过,并挑战了顺序流水线执行的传统做法。我们推导了具有跳过和交换流水线阶段的流水线化的收敛性和吞吐量约束(指南)。基于这些约束,我们提出了SkipPipe,这是第一个部分流水线框架,旨在减少LLM的端到端训练时间,同时保持收敛性。SkipPipe的核心是一个路径调度算法,该算法优化单个微批次的路径,并减少分布式节点上的空闲时间(由于微批次冲突),同时符合给定的阶段跳过比率。我们在从500M到8B参数的LLaMa模型上,在多达20个节点上广泛评估了SkipPipe。我们的结果表明,与全流水线相比,SkipPipe将训练迭代时间减少了高达55%。我们的部分流水线训练还提高了推理期间对层遗漏的抵抗性,在仅运行一半模型时,困惑度仅下降7%。我们的代码可在https://github.com/gensyn-ai/skippipe获取。
英文摘要:
Data and pipeline parallelism are ubiquitous for training of Large Language Models (LLM) on distributed nodes. Driven by the need for cost-effective training, recent work explores efficient communication arrangement for end to end training. Motivated by LLM's resistance to layer skipping and layer reordering, in this paper, we explore stage (several consecutive layers) skipping in pipeline training, and challenge the conventional practice of sequential pipeline execution. We derive convergence and throughput constraints (guidelines) for pipelining with skipping and swapping pipeline stages. Based on these constraints, we propose SkipPipe, the first partial pipeline framework to reduce the end-to-end training time for LLMs while preserving the convergence. The core of SkipPipe is a path scheduling algorithm that optimizes the paths for individual microbatches and reduces idle time (due to microbatch collisions) on the distributed nodes, complying with the given stage skipping ratio. We extensively evaluate SkipPipe on LLaMa models from 500M to 8B parameters on up to 20 nodes. Our results show that SkipPipe reduces training iteration time by up to $55\%$ compared to full pipeline. Our partial pipeline training also improves resistance to layer omission during inference, experiencing a drop in perplexity of only $7\%$ when running only half the model. Our code is available at https://github.com/gensyn-ai/skippipe.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。