MultiPL-MoE：通过混合专家混合实现大语言模型的多编程语言扩展
MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts
AI总结：
本文提出MultiPL-MoE，通过混合专家混合在词元和片段层面优化专家选择，提升大语言模型的多编程语言代码生成能力，实验验证其有效性。
摘要：
尽管大语言模型（LLM）具有出色的代码生成能力，但多语言代码生成仍然极具挑战性。为了解决这一问题，我们旨在在保留最流行的编程语言性能的同时，利用受限的计算资源提升基础LLM的多编程语言（MultiPL）性能。我们将MultiPL视为多种自然语言的一种特殊情况，并提出了一种利用混合专家混合（MoE）的LLM多编程语言扩展方法，称为MultiPL-MoE。具体而言，MultiPL-MoE结合了两个配对的MoE，以在词元（token）和片段（segment）两个层面优化专家选择。词元级MoE是一种标准的升级循环（upcycling）MoE结构，包含一个共享专家和一种新颖的门控权重归一化方法，有助于最终与片段级MoE的融合。片段级MoE引入了两项创新设计，以更好地捕捉编程语言的句法结构和上下文模式：首先，使用滑动窗口将输入词元序列划分为多个片段；然后，采用专家选择路由策略，允许专家选择前k个片段。实验结果证明了MultiPL-MoE的有效性。
英文摘要：
Despite LLMs' excellent code creation capabilities, multilingual code generation remains extremely challenging. To address this, we intent to improve the multi-programming-lingual (MultiPL) performance of the base LLMs while retaining the most popular ones using restricted computational resources. We consider MultiPL to be a special case of multiple natural languages and propose a MultiPL extension of LLMs utilizing a hybrid mixture of experts (MoE), called MultiPL-MoE. Specifically, MultiPL-MoE combines two paired MoEs to optimize expert selection at both the token and segment levels. The token-level MoE is a standard upcycling MoE structure with a shared expert and a novel gate weight normalization approach that aids in the final fusion with the segment-level MoE. The segment-level MoE incorporates two innovative designs to better capture the syntactic structure and contextual patterns of programming languages: First, using a sliding window to partition the input token sequence into multiple segments; Then, adopting an expert-choice routing strategy that allows experts to select the top-k segments. The results of the experiment proved the effectiveness of MultiPL-MoE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
