揭示多语言MoE模型中的语言路由隔离以实现可解释的子网络适应
Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
AI总结：
本文研究了多语言MoE模型中语言路由隔离现象，提出RISE框架通过识别和适应语言特定的专家子网络，提升低资源语言性能并保持其他语言能力。
摘要：
混合专家（MoE）模型在不同语言上表现出显著的性能差异，但驱动这些差距的内部机制尚不明确。本文系统分析了MoE模型中的专家路由模式，揭示了我们称为语言路由隔离的现象，即高资源和低资源语言倾向于激活 largely disjoint 的专家集合。通过分层分析，我们进一步显示路由模式在模型深度上呈现出逐层收敛-发散的模式。基于这些发现，我们提出了RISE（Routing Isolation-guided Subnetwork Enhancement）框架，利用路由隔离来识别和适应语言特定的专家子网络。RISE应用三元选择策略，使用特异性分数在浅层和深层识别语言特定的专家，使用重叠分数在中间层选择通用专家。通过仅训练选定的子网络并冻结其他参数，RISE显著提升了低资源语言的性能，同时保持其他语言的能力。在10种语言上的实验表明，RISE在目标语言上实现了高达10.85%的F1提升，同时最小化了跨语言退化。
英文摘要：
Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we conduct a systematic analysis of expert routing patterns in MoE models, revealing a phenomenon we term Language Routing Isolation, in which high- and low-resource languages tend to activate largely disjoint expert sets. Through layer-stratified analysis, we further show that routing patterns exhibit a layer-wise convergence-divergence pattern across model depth. Building on these findings, we propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expert subnetworks. RISE applies a tripartite selection strategy, using specificity scores to identify language-specific experts in shallow and deep layers and overlap scores to select universal experts in middle layers. By training only the selected subnetwork while freezing all other parameters, RISE substantially improves low-resource language performance while preserving capabilities in other languages. Experiments on 10 languages demonstrate that RISE achieves target-language F1 gains of up to 10.85% with minimal cross-lingual degradation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
