arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.07900cs.SEcs.AI
收藏

重新思考基于LLM的软件工程代理生成测试的价值

Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

Singapore Management University(新加坡管理大学);Shanghai Jiao Tong University(上海交通大学)

Zhi Chen, Zhensu Sun, Yuling Shi, Chao Peng, Xiaodong Gu, David Lo, Lingxiao Jiang

AI总结:

研究探讨了LLM软件工程代理生成测试的有效性,发现测试主要作为反馈渠道,而非提升问题解决效果。

摘要:

大型语言模型(LLM)代码代理通过迭代编辑代码、调用工具和验证候选补丁来解决仓库级问题。在这些工作流中,代理常即兴编写测试,但其价值尚不明确。例如,GPT-5.2几乎不编写新测试却能实现与顶级代理相当的性能。这引发核心问题:此类测试是否真正提升问题解决,还是仅模仿熟悉软件开发实践而消耗交互预算?为更好地理解代理编写测试的作用,我们分析了六个强LLM在SWE-bench Verified上生成的轨迹。结果表明,测试编写普遍存在,但同一体系内解决和未解决任务的测试编写频率相似。当编写测试时,它们主要作为观测反馈渠道,揭示价值的打印语句出现频率远高于基于断言的检查。基于这些见解,我们通过修改四个模型的提示以增加或减少测试编写,发现测试编写量的变化对最终结果无显著影响。总体而言,这些结果表明当前代理编写测试的做法改变了流程并增加了成本,而未显著提升最终任务结果。

英文摘要:

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. This raises a central question: how does agent-written testing relate to task outcomes and resource use during repository-level issue resolution? To better understand the role of agent-written tests, we analyze 5,000 recorded agent runs produced by six strong LLMs on SWE-bench Verified using the lightweight mini-SWE-agent scaffold. Our results show that test writing is common for most models. GPT-5.2 is an exception: it rarely writes new test files while achieving a resolution rate close to the top-ranked model. Test-writing frequency varies widely across models. Within a given model, it differs less between resolved and unresolved tasks. When tests are written, their files contain substantially more statements that print runtime values than assertions that check conditions. We then compare baseline prompts with revised prompts that encourage or discourage writing new test files for four models. Agents start or stop writing test files as requested on 36.8%-93.2% of tasks. Resolved-task totals differ by 0-13 out of 500 per model, with no statistically significant difference, while API calls or token use differ substantially for three models. Similar resolution totals can therefore hide substantial differences in how agents test and how many resources they use.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑