arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07436cs.PL
收藏

McEval:大规模多语言代码评估

McEval: Massively Multilingual Code Evaluation

Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, Zekun Wang, Boyang Wang, Xianjie Wu, Bing W… 展开作者

Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, Zekun Wang, Boyang Wang, Xianjie Wu, Bing Wang, Tongliang Li, Liqun Yang, Sufeng Duan, Zhoujun Li

AI总结:

本文提出覆盖40种编程语言、含16K测试样本的多语言代码评估基准McEval及指令语料McEval-Instruct,并训练多语言编码器mCoder,实验揭示开源与闭源模型在多语言代码任务上仍存在显著差距。

摘要:

代码大语言模型(LLMs)在代码理解、补全和生成任务中展现出显著进展。由一系列代码挑战及相应测试用例构成的编程基准,是评估不同LLMs在这些任务中能力的标准。然而,现有基准大多主要关注Python,且仍局限于少数语言,其中其他语言是从Python样本翻译而来(例如MultiPL-E),降低了数据多样性。为了进一步推动代码LLMs的研究,我们提出了一个覆盖40种编程语言(McEval)、包含16K测试样本的大规模多语言代码基准,大幅拓展了代码LLMs在多语言场景中的能力边界。该基准包含具有挑战性的代码补全、理解和生成评估任务,并配有精心构建的大规模多语言指令语料库McEval-Instruct。此外,我们引入了一个在McEval-Instruct上训练的高效多语言编码器mCoder,以支持多语言编程语言生成。在McEval上的大量实验结果表明,在众多语言中,开源模型与闭源LLMs(例如GPT系列模型)之间仍存在艰难的差距。指令语料库、评估基准和排行榜可在\url{https://mceval.github.io/}获取。

英文摘要:

Code large language models (LLMs) have shown remarkable advances in code understanding, completion, and generation tasks. Programming benchmarks, comprised of a selection of code challenges and corresponding test cases, serve as a standard to evaluate the capability of different LLMs in such tasks. However, most existing benchmarks primarily focus on Python and are still restricted to a limited number of languages, where other languages are translated from the Python samples (e.g. MultiPL-E) degrading the data diversity. To further facilitate the research of code LLMs, we propose a massively multilingual code benchmark covering 40 programming languages (McEval) with 16K test samples, which substantially pushes the limits of code LLMs in multilingual scenarios. The benchmark contains challenging code completion, understanding, and generation evaluation tasks with finely curated massively multilingual instruction corpora McEval-Instruct. In addition, we introduce an effective multilingual coder mCoder trained on McEval-Instruct to support multilingual programming language generation. Extensive experimental results on McEval show that there is still a difficult journey between open-source models and closed-source LLMs (e.g. GPT-series models) in numerous languages. The instruction corpora, evaluation benchmark, and leaderboard are available at \url{https://mceval.github.io/}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑