arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.25399cs.SE
收藏

CoRE:超越输出预测的细粒度代码推理基准

CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction

School of Software Technology, Zhejiang University(浙江大学软件学院);Chinese University of Hong Kong(香港中文大学);Fudan University(复旦大学);Yale University(耶鲁大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织数据61);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新区(滨江)区块链与数据安全研究所)

Jun Gao, Yun Peng, Qian Qiao, Changhai Zhou, Yuhua Zhou, Shiyang Zhang, Shichao Weng, Zhenchang Xing, Xiaoxue Ren

AI总结:

CoRE基准通过实现不变性和过程透明性评估代码推理,揭示大语言模型在等价实现间存在显著鲁棒性差距,并发现模型可能通过表面执行达到正确输出,表明仅评估输出不足以衡量代码推理能力。

摘要:

尽管在代码生成任务上表现出色,但尚不清楚大型语言模型(LLMs)是否真正对代码执行进行推理。现有的代码推理基准主要在单一标准实现下评估最终输出的正确性,留下了两个关键方面未被探索:(1)LLMs是否能保持对功能等价实现的一致性,以及(2)LLMs是否能准确推理中间执行状态。我们引入了\textbf{CoRE},一个评估代码推理的\textbf{Co}de \textbf{Re}asoning基准,通过\textbf{实现不变性}和\textbf{过程透明性}来评估代码推理。对八个前沿LLMs的广泛评估揭示了两个基本限制。首先,模型在等价实现上表现出显著的\textbf{鲁棒性差距},性能在不同实现间变化显著。其次,我们观察到\textbf{表面执行},即模型通过不正确推理中间执行状态而达到正确最终输出。这些发现表明,仅评估输出不足以衡量代码推理能力,CoRE作为评估鲁棒且忠实代码推理的必要基准而被提出。\footnote{数据和代码可在https://github.com/ZJUSig/CoRE获取。}

英文摘要:

Despite strong performance on code generation tasks, it remains unclear whether large language models (LLMs) genuinely reason about code execution. Existing code reasoning benchmarks primarily evaluate final output correctness under a single canonical implementation, leaving two critical aspects underexplored: (1) whether LLMs can maintain consistency to functionally equivalent implementations, and (2) whether LLMs can accurately reason about intermediate execution states. We introduce \textbf{CoRE}, a \textbf{Co}de \textbf{Re}asoning benchmark that evaluates code reasoning through \textbf{implementation invariance} and \textbf{process transparency}. Extensive evaluations on eight frontier LLMs reveal two fundamental limitations. First, models exhibit a substantial \textbf{robustness gap}, with performance varying significantly across equivalent implementations. Second, we observe \textbf{superficial execution}, where models arrive at correct final outputs without correctly reasoning about intermediate execution states. Together, these findings demonstrate that output-only evaluations are insufficient for assessing code reasoning and position CoRE as a necessary benchmark for evaluating robust and faithful code reasoning.\footnote{Data and code are available at https://github.com/ZJUSig/CoRE.}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑