DeAR：通过细粒度All-Reduce流水线加速分布式深度学习
DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining
AI总结：
针对现有分布式深度学习通信调度存在启动延迟高、性能次优的问题，提出将all-reduce拆分为两步的DeAR算法及张量融合方案，可同时重叠反向与前馈计算，在64-GPU集群上实现最高83%和15%的加速。
摘要：
通信调度已被证明可有效加速分布式训练，它能让all-reduce通信与反向传播计算重叠执行，这一技术已在主流分布式深度学习框架中被广泛采用。但该方法存在两个根本问题：（1）每次all-reduce操作的启动延迟过高，且与工作节点数量成正比；（2）由于下一轮迭代中前馈计算的依赖关系与同步要求，其训练性能仅能达到次优水平。我们提出了一种新型调度算法DeAR，它将all-reduce原语拆分为两个连续操作，可同时与反向传播和前馈计算重叠，且无需额外通信。我们还设计了一种实用的张量融合算法以进一步提升训练性能。在5个主流模型上的实验结果表明，在配备10Gb/s以太网和100Gb/s InfiniBand互连的64-GPU集群上，DeAR相比现有最优方案分别实现了最高83%和15%的训练加速比。
英文摘要：
Communication scheduling has been shown to be effective in accelerating distributed training, which enables all-reduce communications to be overlapped with backpropagation computations. This has been commonly adopted in popular distributed deep learning frameworks. However, there exist two fundamental problems: (1) excessive startup latency proportional to the number of workers for each all-reduce operation; (2) it only achieves sub-optimal training performance due to the dependency and synchronization requirement of the feed-forward computation in the next iteration. We propose a novel scheduling algorithm, DeAR, that decouples the all-reduce primitive into two continuous operations, which overlaps with both backpropagation and feed-forward computations without extra communications. We further design a practical tensor fusion algorithm to improve the training performance. Experimental results with five popular models show that DeAR achieves up to 83% and 15% training speedup over the state-of-the-art solutions on a 64-GPU cluster with 10Gb/s Ethernet and 100Gb/s InfiniBand interconnects, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
