arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.00566cs.SEcs.CL
收藏

CodeBenchGen:创建可扩展的基于执行的代码生成基准

CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks

Carnegie Mellon University(卡内基梅隆大学);Shanghai Jiao Tong University(上海交通大学)

Yiqing Xie, Alex Xie, Divyanshu Sheth, Pengfei Liu, Daniel Fried, Carolyn Rose

AI总结:

本文提出CodeBenchGen框架,利用大语言模型将自然代码片段自动沙箱化为含测试用例的评估示例,并构建包含1,931个示例的Exec-CSN数据集,以支持跨场景的基于执行的代码生成评估。

摘要:

为了充分测试现代代码生成系统,评估基准必须执行并测试系统生成的代码。然而,这些执行和测试要求在很大程度上将基准限制在代码易于执行或具有人工编写测试的场景中。为了促进跨多样化场景的代码生成系统评估,我们提出了CodeBenchGen,一个从自然出现的代码源创建可扩展的基于执行的基准的框架。具体而言,我们利用一个大语言模型(LLM)将任意代码片段沙箱化为评估示例,包括用于基于执行的评估的测试用例。我们通过创建一个数据集Exec-CSN来说明我们框架的实用性,该数据集包含1,931个示例,涉及293个库,这些示例从取自CodeSearchNet数据集的367个GitHub仓库中的代码转换而来。为了证明Exec-CSN中示例的可解性,我们展示了一项人类研究,表明81.3%的示例可以被人类解决,61%被评定为“需要努力才能解决”。我们在开源和专有模型上进行了代码生成实验,并分析了人类和模型的性能。我们在以下地址提供代码和数据:https://github.com/yiqingxyq/CodeBenchGen。

英文摘要:

To adequately test modern code generation systems, evaluation benchmarks must execute and test the code generated by the system. However, these execution and testing requirements have largely limited benchmarks to settings where code is easily executable or has human-written tests. To facilitate evaluation of code generation systems across diverse scenarios, we present CodeBenchGen, a framework to create scalable execution-based benchmarks from naturally occurring code sources. Specifically, we leverage a large language model (LLM) to sandbox arbitrary pieces of code into evaluation examples, including test cases for execution-based evaluation. We illustrate the usefulness of our framework by creating a dataset, Exec-CSN, which includes 1,931 examples involving 293 libraries converted from code in 367 GitHub repositories taken from the Code- SearchNet dataset. To demonstrate the solvability of examples in Exec-CSN, we present a human study demonstrating that 81.3% of the examples can be solved by humans and 61% are rated as "requires effort to solve". We conduct code generation experiments on open-source and proprietary models and analyze the performance of both humans and models. We provide code and data at: https://github.com/yiqingxyq/CodeBenchGen.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑