混合聚类专家：推进指令微调中的专家专业化与泛化能力
Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
AI总结：
针对指令微调中输入异质性导致专家专业化不足的问题，提出双阶段路由机制的混合聚类专家模型，通过序列级分组与词元级激活提升专家专业化与泛化能力。
摘要：
稀疏混合专家架构通过在不按比例增加计算成本的情况下条件性激活子模块，已成为一种高度可扩展的解决方案。然而，提升专家专业化以增强性能和泛化能力，对于MoE仍是一项挑战，尤其是在以输入高度异质性为特征的指令微调场景中。在本研究中，我们提出了混合聚类专家以通过双阶段路由机制解决此局限。该机制的第一阶段基于序列级特征执行专家组路由，而第二阶段在组内于词元级别激活top-$k$专家。该方法能基于知识需求有效划分异质输入，鼓励专家组专业化，同时保持词元级路由的优势。我们在一系列综合基准上评估了MoCE，证明了其相对强基线的一致优越性及增强的泛化能力。详细分析进一步突显了MoCE的鲁棒性与有效性。
英文摘要：
A sparse Mixture-of-Experts (MoE) architecture has emerged as a highly scalable solution by conditionally activating sub-modules without a proportional increase in computational costs. However, improving expert specialization to enhance performance and generalization remains a challenge for MoE, especially in instruction tuning scenarios characterized by significant input heterogeneity. In this work, we propose the Mixture-of-Clustered-Experts (MoCE) to address this limitation through a dual-stage routing mechanism. The first stage in the mechanism performs expert group routing based on sequence-level features, while the second stage activates the top-$k$ experts within the group at the token level. This approach enables the effective partitioning of heterogeneous inputs based on their knowledge requirements, encouraging expert group specialization while maintaining the advantages of token-level routing. We evaluate MoCE across a comprehensive set of benchmarks, demonstrating its consistent superiority over strong baselines and its enhanced generalization capabilities. Detailed analysis further highlights the robustness and effectiveness of MoCE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
