迈向对MoE设计选择的实证理解
Towards an empirical understanding of MoE design choices
AI总结：
本文系统评估MoE设计选择对性能的影响，发现学习式路由与随机冻结路由器性能相当，并揭示序列级路由与Token级路由在专家专业化上的差异。
摘要：
在本研究中，我们系统地评估了Mixture of Experts (MoEs)中常见设计选择对验证性能的影响，揭示了在token级别和序列级别上的不同影响。我们还提供了实证证据，表明学习得到的路由器和冻结的、随机初始化的路由器之间性能相当，这表明学习式路由可能并非必要。我们的研究进一步揭示，序列级路由可能导致主题特定的弱专家专业化，这与Token级路由中观察到的语法专业化形成对比。
英文摘要：
In this study, we systematically evaluate the impact of common design choices in Mixture of Experts (MoEs) on validation performance, uncovering distinct influences at token and sequence levels. We also present empirical evidence showing comparable performance between a learned router and a frozen, randomly initialized router, suggesting that learned routing may not be essential. Our study further reveals that Sequence-level routing can result in topic-specific weak expert specialization, in contrast to syntax specialization observed with Token-level routing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
