MoE++：利用零计算专家加速混合专家方法
MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
AI总结：
本文提出MoE++异构混合专家框架，通过引入零计算专家（零、复制、常数专家）实现丢弃、跳过和替换操作，在降低计算开销、提升性能与优化部署的同时，实现1.1-2.1倍专家前向吞吐量提升。
摘要：
在本研究中，我们旨在同时提升混合专家（Mixture-of-Experts, MoE）方法的有效性与效率。为此，我们提出了MoE++，这是一个通用的异构MoE框架，集成了前馈神经网络（FFN）和零计算专家。具体而言，我们引入了三种零计算专家：零专家、复制专家和常数专家，分别对应丢弃、跳过和替换操作。该设计具有三个关键优势：（i）低计算开销：与原始MoE中对所有token使用统一混合机制不同，MoE++允许每个token与动态数量的FFN交互、通过常数向量调整，甚至完全跳过MoE层。（ii）高性能：通过让简单token利用较少的FFN专家，MoE++使更多专家能专注于困难token，从而释放比原始MoE更大的性能潜力。（iii）部署友好：鉴于零计算专家参数量可忽略不计，我们可以在每个GPU上部署所有零计算专家，从而消除FFN专家分布在不同GPU上带来的显著通信开销和专家负载不均问题。此外，我们利用门控残差，使每个token在选择合适专家时能考虑前一层所走的路径。大量实验结果表明，MoE++在取得更优性能的同时，与同等规模的原始MoE模型相比，实现了1.1-2.1倍的专家前向吞吐量，为开发先进且高效的MoE相关模型奠定了坚实基础。
英文摘要：
In this work, we aim to simultaneously enhance the effectiveness and efficiency of Mixture-of-Experts (MoE) methods. To achieve this, we propose MoE++, a general and heterogeneous MoE framework that integrates both Feed-Forward Network~(FFN) and zero-computation experts. Specifically, we introduce three types of zero-computation experts: the zero expert, copy expert, and constant expert, which correspond to discard, skip, and replace operations, respectively. This design offers three key advantages: (i) Low Computing Overhead: Unlike the uniform mixing mechanism for all tokens within vanilla MoE, MoE++ allows each token to engage with a dynamic number of FFNs, be adjusted by constant vectors, or even skip the MoE layer entirely. (ii) High Performance: By enabling simple tokens to utilize fewer FFN experts, MoE++ allows more experts to focus on challenging tokens, thereby unlocking greater performance potential than vanilla MoE. (iii) Deployment Friendly: Given that zero-computation experts have negligible parameters, we can deploy all zero-computation experts on each GPU, eliminating the significant communication overhead and expert load imbalance associated with FFN experts distributed across different GPUs. Moreover, we leverage gating residuals, enabling each token to consider the pathway taken in the previous layer when selecting the appropriate experts. Extensive experimental results demonstrate that MoE++ achieves better performance while delivering 1.1-2.1x expert forward throughput compared to a vanilla MoE model of the same size, which lays a solid foundation for developing advanced and efficient MoE-related models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
