MoE-LPR：通过带有语言先验路由的混合专家机制实现大型语言模型的多语言扩展
MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing
AI总结：
本文提出MoE-LPR方法，通过两阶段训练（升级改造为MoE架构并引入语言先验路由）增强大型语言模型的多语言能力，有效平衡了语言扩展与防止灾难性遗忘，同时保持推理开销不变。
摘要：
大型语言模型（LLMs）通常以英语为中心，这是由于其预训练数据中语言分布不成比例。通过后预训练增强非英语语言能力通常会导致原有语言能力的灾难性遗忘。先前的方法要么在实现良好扩展的同时伴随严重遗忘，要么遗忘轻微但扩展效果差，这表明在扩展语言能力的同时防止遗忘是一个挑战。在本文中，我们提出了一种名为MoE-LPR（带有语言先验路由的混合专家机制）的方法来缓解这一问题。MoE-LPR采用两阶段训练方法来增强多语言能力。首先，通过升级改造将模型后预训练为混合专家架构，其中所有原始参数被冻结并添加新专家。在此阶段，我们专注于提升扩展语言的能力，而不使用任何原始语言数据。然后，模型使用少于后预训练1%的回放数据复习原始语言知识，在此我们引入语言先验路由以更好地恢复原始语言能力。在多个基准上的评估表明，MoE-LPR优于其他后预训练方法。冻结原始参数保留了原始语言知识，而添加新专家保留了学习能力。使用LPR复习能够有效利用参数内的多语言知识。此外，MoE架构在增加总模型参数的同时保持相同的推理开销。大量实验证明MoE-LPR在提升扩展语言和保留原始语言熟练度方面具有卓越的可扩展性。代码和脚本可在 https://github.com/zjwang21/MoE-LPR.git 免费获取。
英文摘要：
Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of the ability of original languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts (MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR's effectiveness in improving expanded languages and preserving original language proficiency with superior scalability. Code and scripts are freely available at https://github.com/zjwang21/MoE-LPR.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
