用于加速混合专家模型的捷径连接专家并行
Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
AI总结：
提出ScMoE架构与重叠并行策略，解耦通信与计算的顺序依赖实现100%重叠，在保持模型质量的同时显著提升MoE训练和推理速度。
摘要：
专家并行已成为将稀疏门控混合专家（MoE）模型的计算工作负载分布到多个设备上的关键策略，使得处理日益大规模的模型成为可能。然而，专家并行固有的All-to-All通信构成了显著瓶颈，限制了MoE模型的效率。尽管现有的优化方法部分缓解了这一问题，但它们仍受限于通信与计算操作之间的顺序依赖。为了应对这一挑战，我们提出了ScMoE，这是一种结合了重叠并行策略的新型捷径连接MoE架构。ScMoE将通信从其传统的顺序排列中解耦，实现了与计算高达100%的重叠。与流行的top-2 MoE基线相比，ScMoE在训练中实现了1.49倍的加速，在推理中实现了1.82倍的加速。此外，我们的实验和分析表明，ScMoE不仅达到了可比的模型质量，在某些情况下甚至超越了现有方法。
英文摘要：
Expert parallelism has emerged as a key strategy for distributing the computational workload of sparsely-gated mixture-of-experts (MoE) models across multiple devices, enabling the processing of increasingly large-scale models. However, the All-to-All communication inherent to expert parallelism poses a significant bottleneck, limiting the efficiency of MoE models. Although existing optimization methods partially mitigate this issue, they remain constrained by the sequential dependency between communication and computation operations. To address this challenge, we propose ScMoE, a novel shortcut-connected MoE architecture integrated with an overlapping parallelization strategy. ScMoE decouples communication from its conventional sequential ordering, enabling up to 100% overlap with computation. Compared to the prevalent top-2 MoE baseline, ScMoE achieves speedups of 1.49 times in training and 1.82 times in inference. Moreover, our experiments and analyses indicate that ScMoE not only achieves comparable but in some instances surpasses the model quality of existing approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
