HyperMoE：通过专家间迁移实现更好的混合专家模型
HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts
AI总结：
针对混合专家模型中稀疏性与专家知识可用性之间的矛盾，提出基于Hypernetworks的HyperMoE框架，利用未选中专家的信息生成补充模块进行知识迁移，在保持稀疏性的同时显著提升了模型性能。
摘要：
用于语言模型的混合专家模型已被证明在增强模型容量方面是有效的，其通过动态地将每个输入词元路由到特定的专家子集进行处理。尽管取得了成功，但大多数现有方法在平衡稀疏性和专家知识可用性方面面临挑战：通过增加专家知识的使用来提升性能，往往会导致专家选择过程中的稀疏性降低。为了缓解这一矛盾，我们提出了HyperMoE，这是一个基于Hypernetworks构建的新型MoE框架。该框架将MoE的计算过程与多任务学习中的知识迁移概念相结合。基于未选定专家的信息生成的特定模块作为补充信息，这使得未被选中的专家的知识能够在保持选择稀疏性的同时得到利用。我们在多个数据集和骨干网络上的全面实证评估表明，在专家数量相同的条件下，HyperMoE显著优于现有的MoE方法。
英文摘要：
The Mixture of Experts (MoE) for language models has been proven effective in augmenting the capacity of models by dynamically routing each input token to a specific subset of experts for processing. Despite the success, most existing methods face a challenge for balance between sparsity and the availability of expert knowledge: enhancing performance through increased use of expert knowledge often results in diminishing sparsity during expert selection. To mitigate this contradiction, we propose HyperMoE, a novel MoE framework built upon Hypernetworks. This framework integrates the computational processes of MoE with the concept of knowledge transferring in multi-task learning. Specific modules generated based on the information of unselected experts serve as supplementary information, which allows the knowledge of experts not selected to be used while maintaining selection sparsity. Our comprehensive empirical evaluations across multiple datasets and backbones establish that HyperMoE significantly outperforms existing MoE methods under identical conditions concerning the number of experts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
