理解混合专家语言模型中的多语言性：路由机制、专家专业化和逐层引导
Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
AI总结：
本文研究了混合专家语言模型中的多语言处理机制，揭示了路由行为、专家专业化和逐层引导对多语言性能的影响，并提出了一种改进方法以提升多语言表现。
摘要：
混合专家（MoE）架构已显示出强大的多语言能力，但其内部机制 underlying 性能提升和跨语言差异仍不够理解。在本工作中，我们对MoE模型进行了系统分析，考察了跨语言和网络深度的路由行为和专家专业化。我们的分析揭示了多语言处理在MoE模型中高度结构化：路由与语言家族对齐，专家利用遵循清晰的逐层模式，高资源语言依赖共享专家，而低资源语言更多依赖语言专属专家，尽管表现较弱。逐层干预进一步显示，早期和晚期MoE层支持语言特定处理，而中间层作为语言无关容量枢纽。基于这些见解，我们提出了一种路由引导的引导方法，该方法在推理时适应性地引导中间层的路由行为向与主导语言相关的共享专家，导致一致的多语言性能提升，特别是对于语言相关语言对。我们的代码可在https://github.com/conctsai/Multilingualism-in-Mixture-of-Experts-LLMs上获得。
英文摘要：
Mixture-of-Experts (MoE) architectures have shown strong multilingual capabilities, yet the internal mechanisms underlying performance gains and cross-language differences remain insufficiently understood. In this work, we conduct a systematic analysis of MoE models, examining routing behavior and expert specialization across languages and network depth. Our analysis reveals that multilingual processing in MoE models is highly structured: routing aligns with linguistic families, expert utilization follows a clear layerwise pattern, and high-resource languages rely on shared experts while low-resource languages depend more on language-exclusive experts despite weaker performance. Layerwise interventions further show that early and late MoE layers support language-specific processing, whereas middle layers serve as language-agnostic capacity hubs. Building on these insights, we propose a routing-guided steering method that adaptively guides routing behavior in middle layers toward shared experts associated with dominant languages at inference time, leading to consistent multilingual performance improvements, particularly for linguistically related language pairs. Our code is available at https://github.com/conctsai/Multilingualism-in-Mixture-of-Experts-LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
