利用避等待分组平均打破并行随机优化中的（全局）壁垒
Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging
AI总结：
本文提出避等待分组模型平均SGD（WAGMA-SGD），通过子组权重交换和分组allreduce减少全局通信，在保持与Allreduce-SGD相近收敛速率的同时，显著提升大规模深度学习训练吞吐量并缩短求解时间。
摘要：
大规模深度学习主要受通信时间制约。将样本分布到多个节点通常能获得最佳性能，但由于全局信息传播和样本长度不均导致的负载不均衡，会带来扩展性挑战。最先进的去中心化优化器缓解了这一问题，但需要更多迭代次数才能达到与全局通信优化器相同的精度。我们提出了避等待分组模型平均（WAGMA）SGD，这是一种避等待随机优化器，通过子组权重交换减少全局通信。其关键思想是将平均方案的算法改进与分组 allreduce 操作的使用相结合。我们证明了 WAGMA-SGD 的收敛性，并通过实验表明它保持了与 Allreduce-SGD 相似的收敛速率。在评估中，我们在 ImageNet 上训练 ResNet-50；训练用于机器翻译的 Transformer；以及进行大规模导航的深度强化学习。与最先进的去中心化 SGD 变体相比，WAGMA-SGD 显著提高了训练吞吐量（例如，在 1,024 个 GPU 上进行强化学习时提升 2.1 倍），并实现了最快的求解时间（例如，使用最短训练时间获得 Transformer 的最高分数）。
英文摘要：
Deep learning at scale is dominated by communication time. Distributing samples across nodes usually yields the best performance, but poses scaling challenges due to global information dissemination and load imbalance across uneven sample lengths. State-of-the-art decentralized optimizers mitigate the problem, but require more iterations to achieve the same accuracy as their globally-communicating counterparts. We present Wait-Avoiding Group Model Averaging (WAGMA) SGD, a wait-avoiding stochastic optimizer that reduces global communication via subgroup weight exchange. The key insight is a combination of algorithmic changes to the averaging scheme and the use of a group allreduce operation. We prove the convergence of WAGMA-SGD, and empirically show that it retains convergence rates similar to Allreduce-SGD. For evaluation, we train ResNet-50 on ImageNet; Transformer for machine translation; and deep reinforcement learning for navigation at scale. Compared with state-of-the-art decentralized SGD variants, WAGMA-SGD significantly improves training throughput (e.g., 2.1x on 1,024 GPUs for reinforcement learning), and achieves the fastest time-to-solution (e.g., the highest score using the shortest training time for Transformer).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
