arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.01210cs.SEcs.CLcs.LG
收藏

你由 ChatGPT 生成的代码真的正确吗?面向代码生成的大语言模型严格评估

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Nanjing University(南京大学)

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming Zhang

AI总结:

本文提出 EvalPlus 框架,通过 LLM 与变异策略自动大规模扩充测试用例,构建 HumanEval+,严格评估 26 个代码生成 LLM 的真实功能正确性并揭示原有基准排名失真。

摘要:

程序合成长期以来一直受到研究,近期方法则聚焦于直接利用大语言模型(LLM)的能力来生成代码。编程基准包含精心整理的合成问题与测试用例,被用于衡量各种 LLM 在代码合成方面的性能。然而,这些测试用例在数量和质量上都可能存在局限,无法充分评估生成代码的功能正确性。现有基准中的这种局限引出了以下问题:在 LLM 时代,生成的代码是否真的正确?为回答该问题,我们提出 EvalPlus——一个代码合成评估框架,用于对 LLM 合成代码的功能正确性进行严格基准测试。EvalPlus 通过自动测试输入生成器新产生的大量测试用例来扩充给定评估数据集,该生成器同时由基于 LLM 和基于变异的策略驱动。尽管 EvalPlus 具有通用性,我们仍将流行的 HumanEval 基准的测试用例扩展了 80 倍,构建了 HumanEval+。我们对 26 个流行 LLM(例如 GPT-4 和 ChatGPT)进行的广泛评估表明,HumanEval+ 能够发现大量此前未被检测到的、由 LLM 合成的错误代码,使 pass@k 最高降低 19.3%–28.9%。我们还意外发现,测试不充分会导致错误排名。例如,WizardCoder-CodeLlama 和 Phind-CodeLlama 现在在 HumanEval+ 上均优于 ChatGPT,而它们在 HumanEval 上都无法做到这一点。我们的工作不仅表明以往流行的代码合成评估结果无法准确反映 LLM 在代码合成方面的真实性能,也开辟了通过自动化测试改进此类编程基准的新方向。我们已在 https://github.com/evalplus/evalplus 开源工具、增强后的数据集以及所有 LLM 生成的代码,以促进并加速未来面向代码的 LLM 研究。

英文摘要:

Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑