MoE-CT：一种具有抗灾难性遗忘能力的大语言模型训练新方法
MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting
AI总结：
本文提出MoE-CT架构，通过冻结基础LLM参数并附加独立训练的MoE模块，在提升低资源语言能力的同时避免灾难性遗忘，实验证明其优于传统持续训练方法并具备更强的迁移学习能力。
摘要：
大语言模型（LLMs）的出现主要服务于高资源语言，导致低资源语言在性能上存在差距。用于弥合这一差距的传统持续训练（CT）方法在扩展到多语言语境时，往往会损害模型原有的语言能力。针对这一问题，我们提出了一种新颖的MoE-CT架构，该范式创新性地将基础模型的学习与多语言扩展过程分离开来。我们的设计冻结原始LLM参数，从而保障其在高资源语言上的性能，同时附加一个在多样化语言数据集上训练的MoE模块，以增强低资源语言能力。实验表明，我们的方法显著优于传统CT方法，在多语言基准测试中取得了明显提升，同时没有牺牲模型原有的语言性能。此外，我们的MoE-CT框架表现出更强的抗遗忘能力和更优的迁移学习能力。通过保持基础模型的完整性并专注于策略性参数扩展，我们的方法推进了多语言语言建模，并代表了低资源语言纳入LLMs的重要一步，为语言技术的未来研究指明了一个富有成效的方向。
英文摘要：
The advent of large language models (LLMs) has predominantly catered to high-resource languages, leaving a disparity in performance for low-resource languages. Conventional Continual Training (CT) approaches to bridge this gap often undermine a model's original linguistic proficiency when expanding to multilingual contexts. Addressing this issue, we introduce a novel MoE-CT architecture, a paradigm that innovatively separates the base model's learning from the multilingual expansion process. Our design freezes the original LLM parameters, thus safeguarding its performance in high-resource languages, while an appended MoE module, trained on diverse language datasets, augments low-resource language proficiency. Our approach significantly outperforms conventional CT methods, as evidenced by our experiments, which show marked improvements in multilingual benchmarks without sacrificing the model's original language performance. Moreover, our MoE-CT framework demonstrates enhanced resistance to forgetting and superior transfer learning capabilities. By preserving the base model's integrity and focusing on strategic parameter expansion, our methodology advances multilingual language modeling and represents a significant step forward for low-resource language inclusion in LLMs, indicating a fruitful direction for future research in language technologies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
