HMoE：用于语言建模的异构专家混合模型
HMoE: Heterogeneous Mixture of Experts for Language Modeling
AI总结：
本文提出异构专家混合模型HMoE，通过不同大小的专家处理不同复杂度的令牌，并设计训练目标促进小专家激活，以更少参数实现更低损失，超越传统同质MoE。
摘要：
专家混合模型（Mixture of Experts, MoE）通过选择性地激活模型参数的子集，提供了卓越的性能和计算效率。传统上，MoE模型使用同质专家，每个专家具有相同的容量。然而，输入数据复杂性的变化需要具有不同能力的专家，而同质MoE阻碍了有效的专家专业化和高效的参数利用。在本研究中，我们提出了一种新颖的异构专家混合模型（Heterogeneous Mixture of Experts, HMoE），其中专家的大小不同，因此具有不同的容量。这种异构性允许更专业的专家更有效地处理不同的令牌复杂性。为了解决专家激活的不平衡问题，我们提出了一种新颖的训练目标，鼓励更频繁地激活较小的专家，从而提高计算效率和参数利用率。大量实验表明，HMoE以更少的激活参数实现了更低的损失，并在各种预训练评估基准上优于传统的同质MoE模型。代码将在论文被接收后发布。
英文摘要：
Mixture of Experts (MoE) offers remarkable performance and computational efficiency by selectively activating subsets of model parameters. Traditionally, MoE models use homogeneous experts, each with identical capacity. However, varying complexity in input data necessitates experts with diverse capabilities, while homogeneous MoE hinders effective expert specialization and efficient parameter utilization. In this study, we propose a novel Heterogeneous Mixture of Experts (HMoE), where experts differ in size and thus possess diverse capacities. This heterogeneity allows for more specialized experts to handle varying token complexities more effectively. To address the imbalance in expert activation, we propose a novel training objective that encourages the frequent activation of smaller experts, enhancing computational efficiency and parameter utilization. Extensive experiments demonstrate that HMoE achieves lower loss with fewer activated parameters and outperforms conventional homogeneous MoE models on various pre-training evaluation benchmarks. Codes will be released upon acceptance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
