CO2: 实现通信与计算完全重叠的高效分布式训练
CO2: Efficient Distributed Training with Full Communication-Computation Overlap
AI总结:
本研究提出名为CO2的分布式训练方法,通过本地更新与异步通信实现通信计算完全重叠,搭配相关优化技术保障收敛稳定,可兼容ZeRO优化器,在不同带宽集群上均能大幅提升大模型训练的可扩展性。
摘要:
大语言模型的根本成功依赖于大规模分布式训练技术的有效落地。然而,搭建具备高速通信互联能力的大型高性能集群成本极高,仅头部机构有能力负担。本研究旨在降低这一门槛,让带宽有限的集群也能开展大规模训练,推动相关技术普及。我们提出名为CO2的新方法,将本地更新与异步通信引入分布式数据并行训练,从而实现通信与计算的完全重叠。即便在通信带宽极为有限的大型多节点集群上,CO2也能实现高可扩展性。我们还为CO2搭配提出了陈旧度差距惩罚与外动量裁剪技术,以提升其收敛性与训练稳定性。此外,CO2可与成熟的ZeRO系列优化器无缝集成,这类优化器能在大模型训练时降低模型状态的内存消耗。我们提供了收敛性的数学证明,并推导了严格的上界。此外,我们通过涵盖计算机视觉与自然语言处理领域多种任务的大量实际实验验证了研究结果,这些实验证明了在最多包含128张A100 GPU的配置下部署CO2时,其在收敛性、泛化性与可扩展性方面的性能。实验结果凸显,无论是在配备800Gbps RDMA还是80Gbps TCP/IP节点间连接的集群上,CO2都具备大幅提升可扩展性的出色能力。
英文摘要:
The fundamental success of large language models hinges upon the efficacious implementation of large-scale distributed training techniques. Nevertheless, building a vast, high-performance cluster featuring high-speed communication interconnectivity is prohibitively costly, and accessible only to prominent entities. In this work, we aim to lower this barrier and democratize large-scale training with limited bandwidth clusters. We propose a new approach called CO2 that introduces local-updating and asynchronous communication to the distributed data-parallel training, thereby facilitating the full overlap of COmunication with COmputation. CO2 is able to attain a high scalability even on extensive multi-node clusters constrained by very limited communication bandwidth. We further propose the staleness gap penalty and outer momentum clipping techniques together with CO2 to bolster its convergence and training stability. Besides, CO2 exhibits seamless integration with well-established ZeRO-series optimizers which mitigate memory consumption of model states with large model training. We also provide a mathematical proof of convergence, accompanied by the establishment of a stringent upper bound. Furthermore, we validate our findings through an extensive set of practical experiments encompassing a wide range of tasks in the fields of computer vision and natural language processing. These experiments serve to demonstrate the capabilities of CO2 in terms of convergence, generalization, and scalability when deployed across configurations comprising up to 128 A100 GPUs. The outcomes emphasize the outstanding capacity of CO2 to hugely improve scalability, no matter on clusters with 800Gbps RDMA or 80Gbps TCP/IP inter-node connections.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。