Mix-MoE：通过混合专家混合提升大语言模型的多语言机器翻译
Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
AI总结：
提出Mix-MoE框架，通过将MoE层分为语言模型专家和机器翻译专家，并利用傅里叶变换增强路由机制，解决大语言模型在多语言机器翻译微调中的参数干扰问题。
摘要：
大语言模型（LLMs）在多语言机器翻译（MT）中展现出巨大潜力，即使双语监督有限。然而，使用平行语料库微调LLMs带来了主要挑战，即参数干扰。为了解决这些问题，我们提出了Mix-MoE，一个混合专家混合框架，旨在训练LLMs进行多语言MT。我们的框架在两个不同的阶段运行：（1）在单语语料库上使用MoE进行后预训练，以及（2）在平行语料库上使用MoE进行后预训练。关键的是，我们将MoE层分为两个专门的组：语言模型专家（LM专家）和机器翻译专家（MT专家）。LM专家旨在捕获和保留预训练LLM学到的单语知识。另一方面，MT专家专门训练以获取和存储双语翻译知识。此外，为了促进这些专门专家之间的有效交互并利用文本中潜在的结构模式，我们引入了一种由模型表示中的傅里叶变换特征增强的路由机制。实验结果表明，Mix-MoE在多语言MT中表现出色，显著优于现有基线，并在缓解参数干扰方面取得了显著进展。
英文摘要：
Large Language Models (LLMs) have shown great promise in multilingual machine translation (MT), even with limited bilingual supervision. However, fine-tuning LLMs with parallel corpora presents major challenges, namely parameter interference. To address these issues, we propose Mix-MoE, a mixed Mixture-of-Experts framework designed to train LLMs for multilingual MT. Our framework operates in two distinct stages: (1) post-pretraining with MoE on monolingual corpora, and (2) post-pretraining with MoE on parallel corpora. Crucially, we divide the MoE layers into two specialized groups: Language Model Experts (LM Experts) and Machine Translation Experts (MT Experts). LM Experts are designed to capture and retain the monolingual knowledge learned by the pre-trained LLM. MT Experts, on the other hand, are specifically trained to acquire and store bilingual translation knowledge. Furthermore, to facilitate effective interaction between these specialized experts and leverage potential underlying structural patterns in text, we introduce a routing mechanism enhanced by Fourier Transform features derived from model representations. The experimental results demonstrate that Mix-MoE excels in multilingual MT, significantly outperforming existing baselines and showing notable progress in mitigating parameter interference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
