需要多少专家？迈向混合专家架构的最优语义专业化
How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
AI总结：
MASS通过自适应语义专业化和动态路由策略，优化混合专家架构的语义分化，提升模型性能和领域鲁棒性。
摘要：
寻找稀疏混合专家（SMoE）的最佳配置，以最大化专家之间的语义差异，对于充分利用MoE架构的全部潜力至关重要。然而，现有的SMoE框架要么严重依赖超参数调优，要么在调整专家池大小时忽视了在专家之间多样化语义角色的重要性。我们提出了一种名为混合专家用于自适应语义专业化的（MASS）框架，这是一种具有语义意识的MoE框架，用于自适应专家扩展和动态路由。MASS引入了两个关键进展：（i）基于梯度的语义漂移检测器，当现有专家池无法捕捉数据的完整语义多样性时，会提示针对性的专家扩展；（ii）一种适应性路由策略，根据令牌级路由置信度质量动态调整专家使用。我们首先证明，在高度受控的合成设置中，MASS能够可靠地收敛到成本-性能权衡的最佳平衡点，并显著提高了语义专业化。进一步在语言和视觉领域的真实世界数据集上进行的实验证明，MASS在多个强大的MoE基线模型上表现一致，展示了其领域鲁棒性和增强的专家专业化能力。
英文摘要：
Finding the optimal configuration of Sparse Mixture-ofExperts (SMoE) that maximizes semantic differentiation among experts is essential for exploiting the full potential of MoE architectures. However, existing SMoE frameworks either heavily rely on hyperparameter tuning or overlook the importance of diversifying semantic roles across experts when adapting the expert pool size. We propose Mixture-of-Experts for Adaptive Semantic Specialization (MASS), a semanticaware MoE framework for adaptive expert expansion and dynamic routing. MASS introduces two key advancements: (i) a gradient-based semantic drift detector that prompts targeted expert expansion when the existing expert pool lacks capacity to capture the full semantic diversity of the data, and (ii) an integration of adaptive routing strategy that dynamically adjusts expert usage based on token-level routing confidence mass. We first demonstrate that MASS reliably converges to the point of optimal balance between cost-performance trade-off with notably improved sematic specialization in a highly controlled synthetic setup. Further empirical results on real-world datasets across language and vision domains show that MASS consistently outperforms a range of strong MoE baselines, demonstrating its domain robustness and enhanced expert specialization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
