arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.06401cs.SEcs.AIcs.CL
收藏

DevEval:评估实际软件项目中的代码生成能力

DevEval: Evaluating Code Generation in Practical Software Projects

Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Zhi Jin, Hao Zhu, Huanyu Liu, Kaibo Liu, Lecheng Wang, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yihong D… 展开作者

Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Zhi Jin, Hao Zhu, Huanyu Liu, Kaibo Liu, Lecheng Wang, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yihong Dong, Yuqi Zhu, Bin Gu, Mengfei Yang

AI总结:

针对现有代码生成基准与实际项目脱节的问题,提出贴合真实开发场景的DevEval基准,含119个项目的2690个样本,评估5种主流LLM后发现gpt-3.5-turbo最高Pass@1仅42,该基准已开源以助力相关研究。

摘要:

如何评估大语言模型(LLM)的代码生成能力是一个开放性问题。现有许多基准测试与实际软件项目存在脱节,例如程序分布不真实、依赖项不足、项目上下文规模过小等,因此LLM在实际项目中的能力仍不明确。本文提出了一个名为DevEval的新基准,契合开发者在实际项目中的开发经验。DevEval通过严格的流程收集,包含来自119个实际项目的2690个样本,覆盖10个领域。与以往基准相比,DevEval在多个维度贴合实际项目,例如真实的程序分布、充足的依赖项、足够规模的项目上下文。我们在DevEval上评估了5种主流LLM(如gpt-4、gpt-3.5-turbo、CodeLLaMa、StarCoder),揭示了它们代码生成的实际能力,例如实验中gpt-3.5-turbo的最高Pass@1仅为42。我们还讨论了实际项目中代码生成的挑战与未来方向。我们已开源DevEval,希望能推动实际项目中代码生成技术的发展。

英文摘要:

How to evaluate Large Language Models (LLMs) in code generation is an open question. Many benchmarks have been proposed but are inconsistent with practical software projects, e.g., unreal program distributions, insufficient dependencies, and small-scale project contexts. Thus, the capabilities of LLMs in practical projects are still unclear. In this paper, we propose a new benchmark named DevEval, aligned with Developers' experiences in practical projects. DevEval is collected through a rigorous pipeline, containing 2,690 samples from 119 practical projects and covering 10 domains. Compared to previous benchmarks, DevEval aligns to practical projects in multiple dimensions, e.g., real program distributions, sufficient dependencies, and enough-scale project contexts. We assess five popular LLMs on DevEval (e.g., gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder) and reveal their actual abilities in code generation. For instance, the highest Pass@1 of gpt-3.5-turbo only is 42 in our experiments. We also discuss the challenges and future directions of code generation in practical projects. We open-source DevEval and hope it can facilitate the development of code generation in practical projects.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑