利用混合专家模型中的路由动态实现高效语言适配
Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
AI总结：
研究英语中心混合专家模型在多语言持续预训练中的路由动态，发现早期和中间层路由分散且语言无关，最终层出现语言专化，并提出仅更新最终层语言特定和共享专家的参数高效适配策略。
摘要：
混合专家（MoE）模型被广泛用于扩展语言模型，但其专家路由行为和多语言环境下的适配仍未被充分探索。在这项工作中，我们研究了在英语中心的MoE模型上使用多语言语料库进行持续预训练时的多语言路由动态，分析了专家使用如何随语言变化。我们发现，持续的多语言预训练导致早期和中间层出现分散的、与语言无关的路由，而语言专化主要出现在最终层。我们还表明，语言之间的token级词汇重叠在路由方式中起着重要作用。受这些发现启发，我们提出了一种参数高效的适配策略，仅更新最终MoE层中的语言特定和共享专家。在MultiBLiMP和Belebele上的实验表明，我们的方法实现了强大的性能-效率权衡，在更新不到2%参数的情况下，达到了与微调整个最终层相竞争的性能。总体而言，我们的发现揭示了在持续预训练期间MoE中语言专化出现的位置和方式，并为低资源多语言适配提供了实用见解。我们的代码可在https://github.com/aditi184/moe-routing-adaptation获取。
英文摘要：
Mixture-of-Experts (MoE) models are widely used to scale language models, yet their expert routing behavior and adaptation in a multilingual setting remain underexplored. In this work, we study multilingual routing dynamics during continual pre-training of an English-centric MoE model on a multilingual corpus, analyzing how expert usage varies across languages. We find that continual multilingual pre-training leads to diffused, language-agnostic routing in early and middle layers, with language specialization primarily emerging in the final layers. We also show that token-level vocabulary overlap between languages plays an important role in how languages are routed. Motivated by these findings, we propose a parameter-efficient adaptation strategy that updates language-specific and shared experts in the final MoE layers. Experiments on MultiBLiMP and Belebele show that our method achieves a strong performance-efficiency trade-off, attaining competitive performance relative to fine-tuning complete final layers, while updating less than 2% of the parameters. Overall, our findings provide insights into where and how language specialization emerges in MoEs during continual pre-training and provide practical insights for low-resource multilingual adaptation. Our code is available at https://github.com/aditi184/moe-routing-adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
