arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09664cs.SEcs.AIcs.CLcs.PL
收藏

CodeMind: 评估大型语言模型的代码推理能力

CodeMind: Evaluating Large Language Models for Code Reasoning

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Changshu Liu, Yang Chen, Reyhaneh Jabbarvand

AI总结:

本文提出CodeMind框架,通过独立执行推理、规范推理和动态语义推理任务评估LLM的代码推理能力,发现LLM在处理复杂代码时性能下降,且bug修复性能与代码推理任务无关。

摘要:

大型语言模型(LLMs)已被广泛用于自动化编程任务。其能力通过测试或证明生成代码的质量来评估。它们能对代码进行推理的程度是一个关键问题,揭示了它们真实能力的重要见解。本文介绍了CodeMind,一个框架,通过以下显式和隐式代码推理任务来评估LLM的代码推理能力:独立执行推理(IER)、规范推理(SR)和动态语义推理(DSR)。第一项评估LLM模拟给定输入到代码的执行并预测输出的能力(IER)。第二项评估LLM将测试数据的模拟纳入代码生成的能力(SR)。最后,CodeMind评估LLM在仅给定特定输入/输出的情况下理解整体代码语义的能力(DSR)。我们对十个LLM在四个广泛使用的基准上进行的广泛评估显示,LLM根据其大小和训练策略,能够推理某些代码的动态方面。然而,对于复杂度更高、非平凡逻辑和算术运算符、非基本类型和API调用的代码,其性能下降。我们显示这些推理任务以不同方式评估LLM,并且全面的代码推理评估需要它们全部。最后,我们显示LLM在bug修复中的性能与任何代码推理任务无关,并且除了先进的前沿模型外,其他LLM在进行bug修复时并不整合代码推理。

英文摘要:

Large Language Models (LLMs) have been widely used to automate programming tasks. Their capabilities have been evaluated by assessing the quality of generated code through tests or proofs. The extent to which they can reason about code is a critical question revealing important insights about their true capabilities. This paper introduces CodeMind, a framework designed to gauge the code reasoning abilities of LLMs through the following explicit and implicit code reasoning tasks: Independent Execution Reasoning (IER), Specification Reasoning (SR) and Dynamic Semantics Reasoning (DSR). The first evaluates the abilities of LLMs to simulate the execution of given inputs to a code and predict the output (IER). The second assesses the abilities of LLMs to incorporate the simulation of test data in the specification into code generation (SR). Finally, CodeMind evaluates LLMs' abilities to understand overall code semantics only given a specific input/output (DSR). Our extensive evaluation of ten LLMs across four widely used benchmarks using CodeMind shows that LLMs, depending on their size and training strategy, can reason about some dynamic aspects of code. However, their performance drops for code with higher complexity, non-trivial logical and arithmetic operators, non-primitive types, and API calls. We show that these reasoning tasks evaluate LLMs differently, and a comprehensive evaluation of code reasoning requires them all. Finally, we show that the performance of LLMs in bug repair is not correlated with any of the code reasoning tasks, and except for advanced frontier models, other LLMs do not incorporate code reasoning when performing bug repair.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑