分层梯度累积和模块化流水线并行：快速高效地训练大语言模型
Layered gradient accumulation and modular pipeline parallelism: fast and efficient training of large language models
摘要：
Transformer的出现引发了语言模型规模的快速增长，远远超过了硬件的改进。在不久的将来，（密集）Transformers预计将达到万亿参数规模，为此训练需要数千甚至数万个GPU。我们调查了在这种规模及更大规模的商用硬件上进行培训的挑战。特别是，我们分析了分布式训练的不同配置的最短可能的训练时间，利用语言模型的经验缩放定律来估计最佳（临界）批量大小。与流行的观点相反，我们没有发现记忆墙的证据，而是认为真正的限制-除了成本-在于训练时间。除了这个分析，我们介绍了两种新的方法，\textit{layered gradient accumulation}和\textit{modular pipeline parallelism}，这两种方法一起将最短的训练时间缩短了一半。这些方法还减少了数据移动，将网络要求降低到不需要快速InfiniBand连接的程度。这种提高的网络效率也改进了ZeRO优化器引入的方法，将内存使用量减少到可用GPU内存的一小部分。
英文摘要：
The advent of the transformer has sparked a quick growth in the size of language models, far outpacing hardware improvements. (Dense) transformers are expected to reach the trillion-parameter scale in the near future, for which training requires thousands or even tens of thousands of GPUs. We investigate the challenges of training at this scale and beyond on commercially available hardware. In particular, we analyse the shortest possible training time for different configurations of distributed training, leveraging empirical scaling laws for language models to estimate the optimal (critical) batch size. Contrary to popular belief, we find no evidence for a memory wall, and instead argue that the real limitation -- other than the cost -- lies in the training duration. In addition to this analysis, we introduce two new methods, \textit{layered gradient accumulation} and \textit{modular pipeline parallelism}, which together cut the shortest training time by half. The methods also reduce data movement, lowering the network requirement to a point where a fast InfiniBand connection is not necessary. This increased network efficiency also improve on the methods introduced with the ZeRO optimizer, reducing the memory usage to a tiny fraction of the available GPU memory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
