arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.16437cs.SEcs.CL
收藏

使用LLM对程序的运行时行为进行推理:我们还有多远?

Reasoning Runtime Behavior of a Program with LLM: How Far Are We?

School of Software Technology, Zhejiang University(浙江大学软件学院);York University(约克大学);Peking University(北京大学);Zhejiang University(浙江大学)

Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, Xin Xia

AI总结:

本文提出REval框架评估代码大模型在程序执行中的推理能力与一致性,实验表明现有模型在运行时行为推理和增量一致性上表现不佳,亟需加强。

摘要:

代码大语言模型(即 code LLMs)展现出了强大的代码理解与生成能力。为了评估 code LLMs 在各个方面的能力,人们提出了许多基准(例如 HumanEval 和 ClassEval)。代码推理是 code LLMs 最核心的能力之一,但现有的代码推理基准并不充分。通常,它们侧重于预测程序的输入和输出,忽略了对程序执行期间中间行为的评估,以及在执行推理时的逻辑一致性(例如,如果执行路径的预测错误,模型不应给出正确的输出)。为了解决这些问题,本文提出了一个名为 REval 的框架,用于评估 code LLMs 的代码推理能力和程序执行的一致性。我们利用现有的代码基准,并在我们的框架内将其适配为新的基准。我们进行了大规模的实证研究,大多数 LLMs 在运行时行为推理(即平均准确率为 44.4%)和增量一致性评估(即平均 IC 分数为 10.3)上均表现不佳。当前 code LLMs 的评估结果反映出社区迫切需要加强 code LLMs 的代码推理能力。我们的代码、数据和排行榜可在 https://r-eval.github.io 获取。

英文摘要:

Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities. To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval). Code reasoning is one of the most essential abilities of code LLMs, but existing benchmarks for code reasoning are not sufficient. Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior during program execution, as well as the logical consistency (e.g., the model should not give the correct output if the prediction of execution path is wrong) when performing the reasoning. To address these problems, in this paper, we propose a framework, namely REval, for evaluating code reasoning abilities and consistency of code LLMs with program execution. We utilize existing code benchmarks and adapt them to new benchmarks within our framework. A large-scale empirical study is conducted and most LLMs show unsatisfactory performance on both Runtime Behavior Reasoning (i.e., an average accuracy of 44.4%) and Incremental Consistency Evaluation (i.e., an average IC score of 10.3). Evaluation results of current code LLMs reflect the urgent need for the community to strengthen the code reasoning capability of code LLMs. Our code, data, and \newname leaderboard are available at https://r-eval.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑