arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.22129cs.SEcs.AIcs.LG
收藏

SWE-Replay:面向软件工程代理的高效测试时间扩展

SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents

Siebel School of Computing(计算科学系);Data Science, University of Illinois Urbana-Champaign, USA(数据科学,伊利诺伊大学厄巴纳-香槟分校)

Yifeng Ding, Lingming Zhang

AI总结:

SWE-Replay通过重用历史轨迹和动态选择探索或利用策略,实现了高效且可推广的软件工程代理测试时间扩展。

摘要:

测试时间扩展已被广泛采用,以增强大型语言模型(LLM)代理在软件工程(SWE)任务中的能力。然而,标准方法通过从头开始重复采样轨迹是计算上昂贵的。尽管最近的方法试图通过使用专门的价值代理来减轻成本,但它们可能会导致模型校准错误,并且无法推广到现代代理,这些代理能够合成定制的bash脚本作为工具。在本文中,我们介绍了SWE-Replay,这是首个高效的、可推广的测试时间扩展技术,适用于现代代理,且不依赖于可能有噪声的价值估计。SWE-Replay通过重用先前试验中的轨迹来优化扩展过程,动态选择要么从头探索,要么通过在关键中间步骤利用存档经验来利用。这种中间步骤的选择是由仓库探索的潜力和推理重要性驱动的,而不是外部LLM质量估计。我们的评估显示,在SWE-Bench Verified上,SWE-Replay始终优于朴素扩展,成本降低了高达17.4%,同时保持或甚至提高了3.8%的性能。进一步在SWE-Bench Pro和多语言上进行的评估验证了SWE-Replay的可推广性,确立了其作为高效软件工程代理测试时间扩展稳健基础的地位。

英文摘要:

Test-time scaling has been widely adopted to enhance the capabilities of Large Language Model (LLM) agents in software engineering (SWE) tasks. However, the standard approach of repeatedly sampling trajectories from scratch is computationally expensive. While recent methods have attempted to mitigate costs using specialized value agents, they can suffer from model miscalibration and fail to generalize to modern agents that synthesize custom bash scripts as tools. In this paper, we introduce SWE-Replay, the first efficient and generalizable test-time scaling technique for modern agents without reliance on potentially noisy value estimates. SWE-Replay optimizes the scaling process by recycling trajectories from prior trials, dynamically choosing to either explore from scratch or exploit archived experience by branching at critical intermediate steps. This selection of intermediate steps is driven by the potential and reasoning significance of repository exploration, rather than external LLM-based quality estimates. Our evaluation shows that, on SWE-Bench Verified, SWE-Replay consistently outperforms naive scaling, reducing costs by up to 17.4% while maintaining or even improving performance by up to 3.8%. Further evaluation on SWE-Bench Pro and Multilingual validates the generalizability of SWE-Replay, establishing it as a robust foundation for efficient test-time scaling of software engineering agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑