多头混合专家
Multi-Head Mixture-of-Experts
AI总结：
针对SMoE存在的专家激活率低和单token多语义概念分析能力不足的问题，提出多头混合专家（MH-MoE），通过多头机制将token拆分为子token并行交由多专家处理并重组，显著提升专家激活率与上下文理解能力。
摘要：
稀疏混合专家（SMoE）在不显著增加训练和推理成本的情况下扩展了模型容量，但存在以下两个问题：（1）专家激活率低，即仅有一小部分专家被激活以进行优化。（2）缺乏对单个 token 内多个语义概念的细粒度分析能力。我们提出了多头混合专家（MH-MoE），它采用多头机制将每个 token 拆分为多个子 token。这些子 token 随后被分配给一组多样化的专家并行处理，并无缝重新整合回原始 token 形式。多头机制使模型能够共同关注不同专家中各种表示空间的信息，同时显著提升专家激活率，从而加深上下文理解并缓解过拟合。此外，我们的 MH-MoE 实现简单，且与其他 SMoE 优化方法解耦，易于集成到其他 SMoE 模型中以提升性能。在三项任务上的广泛实验结果——以英语为中心的语言建模、多语言语言建模和掩码多模态建模任务——证明了 MH-MoE 的有效性。
英文摘要：
Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activation, where only a small subset of experts are activated for optimization. (2) Lacking fine-grained analytical capabilities for multiple semantic concepts within individual tokens. We propose Multi-Head Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each token into multiple sub-tokens. These sub-tokens are then assigned to and processed by a diverse set of experts in parallel, and seamlessly reintegrated into the original token form. The multi-head mechanism enables the model to collectively attend to information from various representation spaces within different experts, while significantly enhances expert activation, thus deepens context understanding and alleviate overfitting. Moreover, our MH-MoE is straightforward to implement and decouples from other SMoE optimization methods, making it easy to integrate with other SMoE models for enhanced performance. Extensive experimental results across three tasks: English-focused language modeling, Multi-lingual language modeling and Masked multi-modality modeling tasks, demonstrate the effectiveness of MH-MoE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
