arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.16655cs.CLcs.AIcs.SE
收藏

用于代码变更免执行评估的大型语言模型评论器

Large Language Model Critics for Execution-Free Evaluation of Code Changes

University of Texas at Dallas(德克萨斯大学达拉斯分校);AWS AI Labs(亚马逊云科技人工智能实验室)

Aashish Yadavally, Hoan Nguyen, Laurent Callot, Gauthier Guinet

AI总结:

本研究设计了基于LLM的参考感知评论器,通过黄金测试补丁对仓库级代码变更进行免执行的步骤级评估,在预测可执行性和构建状态上显著优于现有方法,并开源了相关库。

摘要:

大型语言模型(LLMs)通过基于LLM的多步智能体工作流,为自动化软件工程任务(如错误修复、功能添加等)提供了一种有前景的方法。然而,现有的评估此类工作流的指标,主要是构建状态以及偶尔的日志分析,在提供评估所做变更质量所需信息方面过于稀疏且有限。在本研究中,我们设计了基于LLM的评论器,以推导出结构良好且严格的、针对仓库级代码变更的免执行中间/步骤级评估代理。重要的是,我们假设可以访问该问题的黄金测试补丁(即参考感知),以评估生成补丁的语义和可执行性。以黄金测试补丁作为参考,我们预测所有编辑位置的可执行性达到了91.6%的F1分数,将其聚合后,我们可以在SWE-bench中84.8%的实例中预测构建状态。特别是,这种以执行为核心的LLM评论器比其他无参考和参考感知的LLM评论器高出38.9%至72.5%。此外,我们展示了这种参考感知框架在比较不同智能体工作流生成的补丁时的有用性。最后,我们开源了为该项目开发的库,该库允许在其他智能体工作流或其他基准测试中进一步使用。源代码可在https://github.com/amazon-science/code-agent-eval获取。

英文摘要:

Large language models (LLMs) offer a promising way forward for automating software engineering tasks, such as bug fixes, feature additions, etc., via multi-step LLM-based agentic workflows. However, existing metrics for evaluating such workflows, mainly build status and occasionally log analysis, are too sparse and limited in providing the information needed to assess the quality of changes made. In this work, we designed LLM-based critics to derive well-structured and rigorous intermediate/step-level, execution-free evaluation proxies for repo-level code changes. Importantly, we assume access to the gold test patch for the problem (i.e., reference-aware) to assess both semantics and executability of generated patches. With the gold test patch as a reference, we predict executability of all editing locations with an F1 score of 91.6%, aggregating which, we can predict the build status in 84.8% of the instances in SWE-bench. In particular, such an execution-focused LLM critic outperforms other reference-free and reference-aware LLM critics by 38.9% to 72.5%. Moreover, we demonstrate the usefulness of such a reference-aware framework in comparing patches generated by different agentic workflows. Finally, we open-source the library developed for this project, which allows further usage for either other agentic workflows or other benchmarks. The source code is available at https://github.com/amazon-science/code-agent-eval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑