arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.15079cs.SE
收藏

通过大型语言模型评估代码执行推理的连贯性与一致性

Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Models

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Changshu Liu, Yang Chen, Reyhaneh Jabbarvand

AI总结:

本文提出CES任务,评估大语言模型在模拟程序执行和编程任务中的推理能力,引入连贯性概念以判断执行逻辑是否符合常识,提出新的度量标准以衡量推理一致性,发现LLM在编程任务中存在路径敏感分析的不足。

摘要:

本文提出CES,一种评估大型语言模型(LLM)在模拟程序执行和利用推理进行编程任务能力的任务。除了测量执行模拟过程中变量预测的正确性,CES引入连贯性概念以确定模拟是否符合常识执行逻辑,即使预测值错误。这使CES能够排除因推理捷径、幻觉或潜在数据泄露导致的可疑正确输出预测。CES还引入了新的度量标准,以测量在不同素路径覆盖下的推理一致性,范围包括强、弱和随机。评估16个LLM(包括三个推理LLM)表明,在HumanEval上81.42%的执行模拟是连贯的,其中46.92%和53.08%的结果是正确和错误的输出预测。前沿LLM如GPT-4和DeepSeek-R1的执行推理最不连贯,主要是由于自然语言捷径。尽管执行模拟相对连贯,LLM在不同测试中的推理性能却不一致,主要为随机(48.87%)或弱(45.37%),这可能解释了它们在需要路径敏感程序分析的编程任务中的不足。我们还比较了CES与bug预测/定位/修复,发现LLM在bug相关任务中几乎不将执行推理纳入分析,其成功主要归因于固有的模式匹配或自然语言捷径,如果不是数据泄露。没有推理,LLM在处理未见bug或不同上下文中的模式时存在泛化威胁。CES可以系统地检验LLM在这些任务中的可疑成功。

英文摘要:

This paper proposes CES, a task to evaluate the abilities of LLMs in simulating program execution and using that reasoning in programming tasks. Besides measuring the correctness of variable predictions during execution simulation, CES introduces the notion of coherence to determine whether the simulation complies with commonsense execution logic, even if the predicted values along the simulations are incorrect. This enables CES to rule out suspiciously correct output predictions due to reasoning shortcuts, hallucinations, or potential data leakage. CES also introduces a novel metric to measure reasoning consistency across tests with the same or different prime path coverage in a spectrum: strong, weak, and random. Evaluating 16 LLMs (including three reasoning LLMs) using CES indicates 81.42% coherent execution simulation on HumanEval, 46.92% and 53.08% of which result in correct and incorrect output predictions. Frontier LLMs such as GPT-4 and DeepSeek-R1 have the most incoherent execution reasoning, mostly due to natural language shortcuts. Despite relatively coherent execution simulation, LLMs' reasoning performance across different tests is inconsistent, mostly random (48.87%) or weak (45.37%), potentially explaining their weakness in programming tasks that require path-sensitive program analysis to succeed. We also compare CES with bug prediction/localization/repair, which intuitively requires control- and data-flow awareness. We observe that LLMs barely incorporate execution reasoning into their analysis for bug-related tasks, and their success is primarily due to inherent abilities in pattern matching or natural language shortcuts, if not data leakage. Without reasoning, there is a threat to the generalizability of LLMs in dealing with unseen bugs or patterns in different contexts. CES can be used to vet the suspicious success of LLMs in these tasks systematically.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑