arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.03065cs.SEcs.AIcs.LG
收藏

CRUXEval:一个用于代码推理、理解和执行的基准

CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Meta AI

Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, Sida I. Wang

AI总结:

提出CRUXEval基准,包含800个Python函数用于评估代码模型的输入输出预测能力,发现现有模型表现不佳且开源闭源差距明显。

摘要:

我们提出了CRUXEval(代码推理、理解和执行评估),一个由800个Python函数(3-13行)组成的基准。每个函数都带有一个输入-输出对,从而引出两个自然任务:输入预测和输出预测。首先,我们提出了一种用于生成我们的执行基准的通用方案,可用于创建该基准的未来变体。其次,我们在我们的基准上评估了二十个代码模型,并发现许多最近在HumanEval上得分很高的模型在我们的基准上没有显示出同样的改进。第三,我们展示了简单的CoT和微调方案可以在我们的基准上提高性能,但远未解决它。最佳设置,带有思维链(CoT)的GPT-4,在输入和输出预测上分别实现了75%和81%的pass@1。相比之下,Code Llama 34B在输入和输出预测上实现了50%和46%的pass@1,突出了开源和闭源模型之间的差距。由于没有模型接近完美通过CRUXEval,我们提供了GPT-4在简单程序上持续失败的示例,作为了解其代码推理能力和改进领域的视角。

英文摘要:

We present CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each function comes with an input-output pair, leading to two natural tasks: input prediction and output prediction. First, we propose a generic recipe for generating our execution benchmark which can be used to create future variation of the benchmark. Second, we evaluate twenty code models on our benchmark and discover that many recent high-scoring models on HumanEval do not show the same improvements on our benchmark. Third, we show that simple CoT and fine-tuning schemes can improve performance on our benchmark but remain far from solving it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75% and 81% on input and output prediction, respectively. In contrast, Code Llama 34B achieves a pass@1 of 50% and 46% on input and output prediction, highlighting the gap between open and closed source models. As no model is close to acing CRUXEval, we provide examples of consistent GPT-4 failures on simple programs as a lens into its code reasoning capabilities and areas for improvement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑