arXiv 2505.225822025-05-28cs.CL

少即是多：通过层级混合专家模型实现LLM的高效多语言扩展

Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts

Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education（大数据与人工智能交通大数据实验室，北京交通大学，教育部）；School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China（计算机科学与技术学院，北京交通大学，北京，中国）；Pattern Recognition Center, WeChat AI, Tencent Inc, China（模式识别中心，微信AI，腾讯公司，中国）

Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, Jie Zhou

备注：: ACL 2025 (Main), 16 pages, 5 figures, 11 tables

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对LLM多语言扩展中参数成本高且易影响旧语言性能的问题，提出LayerMoE层级专家分配算法，利用语言表示相似性按层分配专家并引入分类器指导旧语言token路由，在减少专家数量的同时超越了基线方法。

摘要：

持续为现有的大语言模型（LLMs）扩展新语言是构建强大多语言LLMs的一种有前景但充满挑战的方法。最大的挑战在于让模型持续学习新语言的同时，保持对旧语言的熟练能力。为实现这一目标，近期的研究利用混合专家模型架构，通过添加新专家来扩展新语言，并通过将相应的token路由到原始模型主干（旧专家）来避免对旧语言的灾难性遗忘。尽管直观，这类方法在扩展新语言时参数成本高昂，且仍不可避免地影响旧语言的性能。为解决这些局限性，我们分析了LLMs中不同层的语言特性，并提出了一种层级专家分配算法来确定每层新专家的合适数量。具体而言，我们发现LLMs中的不同层在语言间表现出不同的表示相似性，随后利用该相似性作为为每层分配专家的指标，即相似性越高，专家越少。此外，为进一步减轻对旧语言的遗忘，我们在相似性较高的层上的路由器网络前添加了一个分类器，以指导旧语言token的路由。实验结果表明，在单次扩展设置中，我们的方法以减少60%的专家数量超越了先前的最先进基线；在终身扩展设置中，以减少33.3%的专家数量超越了基线，证明了我们方法的有效性。

英文摘要：

Continually expanding new languages for existing large language models (LLMs) is a promising yet challenging approach to building powerful multilingual LLMs. The biggest challenge is to make the model continuously learn new languages while preserving the proficient ability of old languages. To achieve this, recent work utilizes the Mixture-of-Experts (MoE) architecture to expand new languages by adding new experts and avoid catastrophic forgetting of old languages by routing corresponding tokens to the original model backbone (old experts). Although intuitive, this kind of method is parameter-costly when expanding new languages and still inevitably impacts the performance of old languages. To address these limitations, we analyze the language characteristics of different layers in LLMs and propose a layer-wise expert allocation algorithm (LayerMoE) to determine the appropriate number of new experts for each layer. Specifically, we find different layers in LLMs exhibit different representation similarities between languages and then utilize the similarity as the indicator to allocate experts for each layer, i.e., the higher similarity, the fewer experts. Additionally, to further mitigate the forgetting of old languages, we add a classifier in front of the router network on the layers with higher similarity to guide the routing of old language tokens. Experimental results show that our method outperforms the previous state-of-the-art baseline with 60% fewer experts in the single-expansion setting and with 33.3% fewer experts in the lifelong-expansion setting, demonstrating the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。