Agentless：揭秘基于大语言模型的软件工程智能体
Agentless: Demystifying LLM-based Software Engineering Agents
AI总结：
本研究针对LLM软件工程智能体复杂度高的问题，提出无需智能体的三阶段方法Agentless，在SWE-bench Lite基准上取得32.00%的最高准确率且成本仅0.70美元，还构建了更严谨的评估基准SWE-bench Lite-S。
摘要：
大语言模型（LLM）的最新进展显著推动了软件开发任务的自动化，包括代码合成、程序修复和测试生成。近期，研究人员和行业从业者开发了多种自主LLM智能体，以执行端到端的软件开发任务。这些智能体具备使用工具、运行命令、观察环境反馈以及规划后续行动的能力。然而，这类基于智能体的方法复杂度较高，再加上当前LLM的能力有限，引出了以下问题：我们真的必须采用复杂的自主软件智能体吗？为尝试回答这个问题，我们构建了Agentless——一种无需智能体的自动化软件开发问题解决方法。与基于智能体的方法冗长复杂的设置相比，Agentless采用简单的三阶段流程：定位、修复和补丁验证，无需让LLM决定后续行动或操作复杂工具。我们在流行的SWE-bench Lite基准上的实验结果显示，令人惊讶的是，简单的Agentless与所有现有开源软件智能体相比，既实现了最高性能（32.00%，96个正确修复），又具备低成本（0.70美元）的优势。此外，我们对SWE-bench Lite中的问题进行了人工分类，发现了存在确切真实补丁或问题描述不充分/具有误导性的问题。据此，我们排除了这类有问题的条目，构建了SWE-bench Lite-S，以开展更严谨的评估与比较。我们的工作凸显了当前自主软件开发领域中，一种简单、可解释的技术被忽视的潜力。我们希望Agentless能够帮助重新设定自主软件智能体的基准、起点和研究边界，并为这一关键方向的未来工作提供启发。
英文摘要：
Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless -- an agentless approach to automatically solve software development problems. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents! Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patch or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite-S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the current overlooked potential of a simple, interpretable technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
