MultiCoder:用于低资源代码完成的多语言编程预训练
MultiCoder: Multi-Programming-Lingual Pre-Training for Low-Resource Code Completion
摘要:
代码完成在学术界和工业界都是一个有价值的话题。最近,已经提出了大规模单语言编程(MonoPL)预训练模型来提高代码完成的性能。然而,低资源的程序设计语言(PL)的代码完成是困难的数据驱动范式,而有大量的开发人员使用低资源的PL。另一方面,很少有研究探讨多编程语言(MultiPL)预训练对代码完成的影响,特别是对低资源程序设计语言的影响。为此,我们提出了MultiCoder通过MultiPL预训练和MultiPL混合专家(MoE)层来增强低资源代码完成。我们进一步提出了一种新的PL级MoE路由策略(PL-MoE),以提高所有PL上的代码完成。CodeXGLUE和MultiCC上的实验结果表明:1)所提出的MultiCoder在低资源程序设计语言上的性能显著优于MonoPL基线; 2)PL-MoE模块进一步提升了六种程序设计语言的性能。此外,我们详细分析了所提出的方法的效果,并探讨了我们的方法在各种情况下的有效性。
英文摘要:
Code completion is a valuable topic in both academia and industry. Recently, large-scale mono-programming-lingual (MonoPL) pre-training models have been proposed to boost the performance of code completion. However, the code completion on low-resource programming languages (PL) is difficult for the data-driven paradigm, while there are plenty of developers using low-resource PLs. On the other hand, there are few studies exploring the effects of multi-programming-lingual (MultiPL) pre-training for the code completion, especially the impact on low-resource programming languages. To this end, we propose the MultiCoder to enhance the low-resource code completion via MultiPL pre-training and MultiPL Mixture-of-Experts (MoE) layers. We further propose a novel PL-level MoE routing strategy (PL-MoE) for improving the code completion on all PLs. Experimental results on CodeXGLUE and MultiCC demonstrate that 1) the proposed MultiCoder significantly outperforms the MonoPL baselines on low-resource programming languages, and 2) the PL-MoE module further boosts the performance on six programming languages. In addition, we analyze the effects of the proposed method in details and explore the effectiveness of our method in a variety of scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。