arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.18482cs.SE
收藏

大型语言模型代码执行推理深度分析工具

A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models

Changshu Liu, Reyhaneh Jabbarvand

AI总结:

本文提出了ExeRScope工具,通过分析代码执行推理框架的结果,深入理解代码属性对LLM代码执行推理的影响,使分析能推广到相似代码,避免繁琐的基准设计。

摘要:

代码执行推理正成为一种新的非功能性指标,用于评估大型语言模型(LLMs)在编程任务中的能力。最先进的框架(CodeMind或REval)和基准(CruxEval)通常侧重于LLM对有限程序上给定代码的输入/输出或中间变量状态/值的预测。然而,目前缺乏对结果进行更深入分析的工具。如果没有这样的工具,关于LLM代码执行推理的观察结果就无法推广到更多数据集,从而阻碍研究社区和从业者设计出具有更好代码执行推理能力的下一代LLM。本文介绍了ExeRScope,这是一系列用于分析代码执行推理框架结果的工具和启发式方法,旨在更好地理解所研究基准中代码属性对代码执行推理的影响。借助此类工具,分析可以推广到具有相似属性的代码上,而无需迫切设计更多基准,因为这是一项繁琐的工作。

英文摘要:

Code Executing Reasoning is becoming a new non-functional metric that assesses the ability of large language models (LLMs) in programming tasks. State-of-the-art frameworks (CodeMind or REval) and benchmarks (CruxEval) usually focus on LLM's prediction of a given code's input/output or intermediate variable states/values on limited programs. However, there is no tool for more in-depth analysis of the results. Without such a tool, the observations about LLM's code execution reasoning cannot be generalized to more datasets, preventing the research community and practitioners from devising the next generation of LLMs with better code execution reasoning abilities. This paper introduces ExeRScope, a series of tools and heuristics to analyze the result of code execution reasoning frameworks to understand better the impact of code properties in the studied benchmarks on the code execution reasoning. With such tooling, analysis can be generalized to code with similar properties without the urgent need to design more benchmarks, which is a cumbersome effort.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑