arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.14868cs.LGcs.CL
收藏

代码生成模型的多语言评估

Multi-lingual Evaluation of Code Generation Models

AWS AI Labs(亚马逊云科技人工智能实验室)
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonu… 展开作者

Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, Bing Xiang

摘要:

我们提出了关于评估代码生成模型的新基准:MBXP和多语言HumanEval以及MathQA-X。这些数据集涵盖10多种程序设计语言,并使用可扩展的转换框架生成,该框架将提示和测试用例从原始Python数据集转换为目标语言中的相应数据。使用这些基准测试,我们能够评估代码生成模型的性能在多语言的方式,并发现泛化能力的语言模型的领域外的语言,多语言模型的优势,单语言,能力的Few-Shot提示教模型新的语言,和zero-shot翻译能力,即使在单语言设置。此外,我们使用我们的代码生成模型来执行大规模的自举,以获得合成的规范解决方案,在几种语言,可用于其他代码相关的评估,如代码插入,鲁棒性,或总结任务。总的来说,我们的基准测试代表了对语言模型的代码生成能力更深入理解的重要一步。我们在https://github.com/amazon-research/mxeval公开发布我们的代码和数据集。

英文摘要:

We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages and are generated using a scalable conversion framework that transpiles prompts and test cases from the original Python datasets into the corresponding data in the target language. Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages, and zero-shot translation abilities even on mono-lingual settings. Furthermore, we use our code generation model to perform large-scale bootstrapping to obtain synthetic canonical solutions in several languages, which can be used for other code-related evaluations such as code insertion, robustness, or summarization tasks. Overall, our benchmarks represents a significant step towards a deeper understanding of language models' code generation abilities. We publicly release our code and datasets at https://github.com/amazon-research/mxeval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑