分布式训练的通信优化：架构、进展与机遇
Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities
AI总结：
本文介绍了分布式深度神经网络训练的一般架构，从并行化策略、集合通信库和网络三层范式回顾了通信优化研究进展，并提倡垂直与水平等跨层协同设计以进一步优化异构资源潜力。
摘要：
过去几年见证了参数数量不断增长的大规模深度神经网络模型的繁荣。训练此类大规模模型通常需要海量的内存和计算资源，这使得分布式训练成为必然。随着近年来GPU性能的快速提升，计算时间不断缩减，使得通信在整体训练时间中占据了更大的比例。因此，优化分布式训练的通信变得至关重要。在本文中，我们简要介绍了分布式深度神经网络训练的一般架构，并从通信优化的角度分析了并行化策略、集合通信库和网络之间的关系，这构成了一个三层范式。随后，我们回顾了该三层范式内当前具有代表性的研究进展。我们发现，当前三层范式中的各层相对独立，在分布式训练场景中存在丰富的跨层协同优化设计空间。因此，我们提倡将三层范式扩展为五层范式的“垂直”与“水平”协同设计。我们还提倡“内部-间”与“主机-网络”协同设计，以进一步发挥异构资源的潜力。我们希望本文能为分布式训练通信优化的未来研究提供一些启示。
英文摘要：
The past few years have witnessed the flourishing of large-scale deep neural network models with ever-growing parameter numbers. Training such large-scale models typically requires massive memory and computing resources, necessitating distributed training. As GPU performance has rapidly evolved in recent years, computation time has shrunk, making communication a larger portion of the overall training time. Consequently, optimizing communication for distributed training has become crucial. In this article, we briefly introduce the general architecture of distributed deep neural network training and analyze relationships among Parallelization Strategy, Collective Communication Library, and Network from the perspective of communication optimization, which forms a three-layer paradigm. We then review current representative research advances within this three-layer paradigm. We find that layers in the current three-layer paradigm are relatively independent and there is a rich design space for cross-layer collaborative optimization in distributed training scenarios. Therefore, we advocate "Vertical" and "Horizontal" co-designs which extend the three-layer paradigm to a five-layer paradigm. We also advocate "Intra-Inter" and "Host-Net" co-designs to further utilize the potential of heterogeneous resources. We hope this article can shed some light on future research on communication optimization for distributed training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
