混合专家中的多语言路由
Multilingual Routing in Mixture-of-Experts
AI总结：
本研究分析了混合专家模型在处理多语言文本时的路由机制，揭示了中间层的跨语言路由对齐现象，并通过简单干预提升了多语言性能。
摘要：
混合专家（MoE）架构已成为扩展现代大语言模型（LLMs）的关键，但对其稀疏路由动态如何响应多语言数据的理解仍然有限。在本工作中，我们利用平行多语言数据集分析专家路由模式，并展示了具有高度可解释性的分层现象。我们发现，MoE模型在早期和晚期解码器层中以语言特定的方式路由令牌，但在中间层中表现出显著的跨语言路由对齐，这与在密集LLMs中观察到的参数共享趋势相似。特别是，我们揭示了模型在某种语言中的表现与其在这些层中如何将令牌路由到英语之间的清晰、强烈的相关性。超越相关性，我们探索了推理时间干预，以诱导更高的跨语言路由对齐。我们介绍了一种方法，通过促进在英语中频繁激活的中间层任务专家来引导路由器，并且它成功地提高了多语言性能。这些1-2%的提升在两个评估任务、三个模型和15种以上的语言中表现得非常一致，尤其是在这些简单的干预措施可以覆盖经过广泛训练的、最先进的LLMs的路由器的情况下。相比之下，干预非中间层或针对多语言专用专家的干预仅导致性能下降。总的来说，我们展示了多个发现，解释了MoEs如何处理非英语文本，并证明了泛化受到模型在所有语言中利用语言通用专家的能力的限制。
英文摘要：
Mixture-of-Experts (MoE) architectures have become the key to scaling modern LLMs, yet little is understood about how their sparse routing dynamics respond to multilingual data. In this work, we analyze expert routing patterns using parallel multilingual datasets and present highly interpretable layer-wise phenomena. We find that MoE models route tokens in language-specific ways in the early and late decoder layers but exhibit significant cross-lingual routing alignment in middle layers, mirroring parameter-sharing trends observed in dense LLMs. In particular, we reveal a clear, strong correlation between a model's performance in a given language and how similarly its tokens are routed to English in these layers. Extending beyond correlation, we explore inference-time interventions that induce higher cross-lingual routing alignment. We introduce a method that steers the router by promoting middle-layer task experts frequently activated in English, and it successfully increases multilingual performance. These 1-2% gains are remarkably consistent across two evaluation tasks, three models, and 15+ languages, especially given that these simple interventions override routers of extensively trained, state-of-the-art LLMs. In comparison, interventions outside of the middle layers or targeting multilingual-specialized experts only yield performance degradation. Altogether, we present numerous findings that explain how MoEs process non-English text and demonstrate that generalization is limited by the model's ability to leverage language-universal experts in all languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
