GatePro：混合专家模型的无参数专家选择优化
GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
AI总结：
GatePro是一种无参数方法，通过识别相似专家对并引入局部竞争机制，直接促进专家选择多样性，减少冗余计算，提升MoE模型的有效容量和性能。
摘要：
现代大型语言模型利用混合专家（Mixture-of-Experts, MoE）架构实现高效扩展，但面临一个关键挑战：功能相似的专家经常被同时选中，造成冗余计算并限制有效模型容量。现有的辅助平衡损失方法改善了令牌分布，但未能解决潜在的专家多样性问题。我们提出GatePro，一种新颖的无参数方法，直接促进专家选择多样性。GatePro识别最相似的专家对并引入局部竞争机制，在保持自然专家专业化的同时防止冗余专家共同激活。我们的综合评估证明了GatePro在多种模型规模和基准上的有效性。分析表明，GatePro能够增强专家多样性，使专家发展出更独特和互补的能力，避免功能冗余。该方法可在任何训练阶段热插拔部署，无需额外可学习参数，为提升MoE有效性提供了实用解决方案。
英文摘要：
Modern large language models leverage Mixture-of-Experts (MoE) architectures for efficient scaling, but face a critical challenge: functionally similar experts are often selected simultaneously, creating redundant computation and limiting effective model capacity. Existing auxiliary balance loss methods improve token distribution but fail to address the underlying expert diversity problem. We introduce GatePro, a novel parameter-free method that directly promotes expert selection diversity. GatePro identifies the most similar expert pairs and introduces localized competition mechanisms, preventing redundant expert co-activation while maintaining natural expert specialization. Our comprehensive evaluation demonstrates GatePro's effectiveness across model scales and benchmarks. Analysis demonstrates GatePro's ability to achieve enhanced expert diversity, where experts develop more distinct and complementary capabilities, avoiding functional redundancy. This approach can be deployed hot-swappable during any training phase without additional learnable parameters, offering a practical solution for improving MoE effectiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
