arXiv 2606.258212026-06-25cs.CLcs.AI

SARA: 通过语义锚定路由对齐解锁混合专家模型中的多语言知识

SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment

TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China（天津大学计算机科学与技术学院TJUNLP实验室）；Alibaba Group, China（阿里巴巴集团）

Tianyu Dong, Yangyang Liu, Jiang Zhou, Xinwei Wu, Xiaohu Zhao, Hao Wang, Heng Liu, Linlong Xu, Longyue Wang, Weihua Luo, Shaolin Zhu, Deyi Xiong

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对低资源语言在稀疏MoE架构中路由不一致的问题，提出SARA框架，利用对称JS散度约束对齐多语言输入与高资源语义锚点的路由分布，提升低资源语言性能。

摘要：

稀疏混合专家（MoE）架构作为一种在参数可扩展性和计算效率之间取得战略平衡的范式，日益具有影响力。然而，低资源语言由于缺乏高质量训练数据，其令牌往往被路由到与高资源输入主要激活的专家不同的专家，这限制了跨语言专家共享。这种跨语言路由差异进而阻碍了它们在多语言环境中的有效性。为了解决这个问题，我们提出了SARA（语义锚定路由对齐），一个旨在将高资源语言作为锚点的专门能力迁移到低资源语言的框架。SARA使用对称Jensen-Shannon（JS）散度约束，明确地将多语言输入的路由分布与高资源语义锚点对齐。与在输出logits上操作的传统蒸馏方法不同，SARA直接对齐MoE层的内部路由分布，鼓励跨语言专家选择的机制一致性。我们在2个LLM上进行了实验，涉及5种低资源语言和3个基准测试。实验结果表明，SARA优于标准指令微调，例如在Global-MMLU上，Qwen3-30B-A3B提升0.8%，Phi-3.5-MoE-instruct提升1.2%。进一步分析表明，SARA有效解决了低资源语言的性能瓶颈，为增强稀疏架构中的多语言能力提供了一条可扩展的路径。

英文摘要：

Sparse Mixture-of-Experts (MoE) architectures have emerged as an increasingly influential paradigm as they offer a strategic balance between parameter scalability and computational efficiency. However, low-resource languages, which suffer from a scarcity of high-quality training data, often have their tokens routed to different experts than those predominantly activated by high-resource inputs, which limits cross-lingual expert sharing. This cross-lingual routing divergence consequently hinders their efficacy in multilingual contexts. To address this issue, we propose SARA (Semantically Anchored Routing Alignment), a framework designed to transfer specialized capabilities from high-resource languages as anchors to low-resource languages. SARA explicitly aligns the routing distribution of multilingual inputs with high-resource semantic anchors using a symmetric Jensen-Shannon (JS) divergence constraint. Unlike traditional distillation methods that operate on output logits, SARA directly aligns the internal routing distributions of MoE layers, encouraging mechanistic consistency in expert selection across languages. We conduct experiments on 2 LLMs across 5 low-resource languages and 3 benchmarks. Experiment results demonstrate that SARA outperforms standard instruction tuning, e.g., +0.8% on Qwen3-30B-A3B and +1.2% on Phi-3.5-MoE-instruct on Global-MMLU. Further analyses show that SARA effectively addresses performance bottlenecks in low-resource languages, providing a scalable pathway to enhance multilingual capabilities in sparse architectures.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。