Chimera：利用双向流水线高效训练大规模神经网络
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
AI总结：
本文提出同步双向流水线并行方案Chimera，最多减少50%流水线气泡并均衡激活内存，在2048个GPU上训练13亿参数GPT-2时吞吐量提升1.16至2.34倍。
摘要：
大规模训练大型深度学习模型极具挑战性。本文提出Chimera，一种新颖的流水线并行方案，它结合双向流水线以高效训练大规模模型。Chimera是一种同步方法，因此不会损失精度，比异步方法更有利于收敛。与最新的同步流水线方法相比，Chimera将气泡（bubble）数量最多减少50%；得益于双向流水线的精细调度，Chimera具有更均衡的激活内存消耗。我们在基于Transformer的语言模型上进行了评估。对于在Piz Daint超级计算机的2,048个GPU节点上运行、具有13亿参数的GPT-2模型，Chimera相比最先进的同步和异步流水线方法将训练吞吐量提升了1.16倍至2.34倍。
英文摘要：
Training large deep learning models at scale is very challenging. This paper proposes Chimera, a novel pipeline parallelism scheme which combines bidirectional pipelines for efficiently training large-scale models. Chimera is a synchronous approach and therefore no loss of accuracy, which is more convergence-friendly than asynchronous approaches. Compared with the latest synchronous pipeline approach, Chimera reduces the number of bubbles by up to 50%; benefiting from the sophisticated scheduling of bidirectional pipelines, Chimera has a more balanced activation memory consumption. Evaluations are conducted on Transformer based language models. For a GPT-2 model with 1.3 billion parameters running on 2,048 GPU nodes of the Piz Daint supercomputer, Chimera improves the training throughput by 1.16x-2.34x over the state-of-the-art synchronous and asynchronous pipeline approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
