arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.12147cs.SEcs.AIcs.CL
收藏

评估自主编程代理中的计划合规性

From Plan to Action: How Well Do Agents Follow the Plan?

University of Illinois Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校);IBM(IBM公司)

Shuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, Reyhaneh Jabbarvand

AI总结:

本文系统分析了编程代理在执行任务时对计划的遵循情况,通过16991条轨迹评估不同计划变体的影响,发现计划提醒能提高任务成功率,但不恰当的计划补充可能降低性能。

摘要:

代理旨在通过自主的推理-行动-观察循环消除对特定任务提示的依赖。然而,它们通常被要求遵循特定任务计划以提供指导,例如通过导航、重现、修补和验证阶段解决软件问题。不幸的是,代理实际遵循此类指导计划的程度尚不清楚。没有这样的分析,就无法确定解决方案是通过正确的战略推理还是其他方式(如数据污染或过拟合基准)达成的。本文提出了对编程代理计划合规性进行广泛系统分析的首次研究,考察了SWE-agent在四个LLM上SWE-bench Verified和SWE-bench Pro下的16,991条轨迹,八个计划变体。在没有明确计划的情况下,代理会回退到训练期间内部化的流程,这些流程往往不完整、过拟合或应用不一致。提供标准计划能提高问题解决能力,我们观察到定期计划提醒可以缓解计划违规并提高任务成功率。较差的计划对性能的影响甚至比没有计划更严重。令人惊讶的是,在早期阶段增加额外的任务相关阶段可能会降低性能,特别是当这些阶段与模型内部的问题解决策略不一致时。这些发现突显了一个研究缺口:需要调整训练范式,使模型学会遵循指导计划,而不是在模型中编码任务特定计划。这要求教会模型进行适应性推理和行动,而不是记忆流程。

英文摘要:

Agents are commonly instructed to follow a task-specific plan for guidance. However, it is unknown to what extent agents actually follow instructed plans. Without such an analysis, determining the extent agents comply with a given plan, it is impossible to assess whether a solution was reached through correct strategic reasoning or through other means, e.g., data contamination or overfitting to a benchmark. This paper presents the first extensive, systematic analysis of plan compliance in programming agents, examining 21,120 trajectories from SWE-agent across four LLMs on SWE-bench Verified and SWE-bench Pro under eight plan variations. Without an explicit plan, agents fall back on internalized workflows during training, which are often incomplete, overfit, or inconsistently applied. Providing the standard plan improves issue resolution, and we observe that periodic plan reminders can mitigate plan violations and improve task success. A subpar plan hurts performance even more than no plan at all. Surprisingly, inserting additional task-relevant phases in the early stage can degrade performance, particularly when these phases do not align with the model's internal problem-solving strategy. These findings call for fine-tuning paradigms that teach models to follow instructed plans, rather than encoding task-specific plans in them, so that they reason and act adaptively, rather than memorizing workflows.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑