arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.04874cs.CL
收藏

Agent规划基准:LLM Agent规划能力的诊断框架

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents

Tongji University(同济大学);Shanghai AI Laboratory(上海人工智能实验室);Harbin Institute of Technology(哈尔滨工业大学);Fudan University(复旦大学);Skywork AI;University of California, Santa Cruz(加州大学圣克鲁兹分校);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学)

Haoyu Sun, Wenxuan Wang, Mingyang Song, Jujie He, Weinan Zhang, Yang Liu, Yang Yang, Yu Cheng

AI总结:

提出Agent规划基准(APB),通过4209个多模态案例和五个设置,诊断LLM Agent在长程规划、工具噪声鲁棒性、校准拒绝和推理时改进方面的系统弱点。

摘要:

规划是LLM Agent的核心:在行动之前,Agent必须分解目标、选择工具、推理约束并决定任务何时不可行。然而,现有的Agent评估通常只报告端到端的成功率,使得难以判断失败源于规划还是执行。我们引入了 extbf{Agent规划基准(APB)},一个针对规划的诊断基准,包含22个领域和五个设置下的4209个多模态案例,涵盖整体规划、反馈条件逐步规划以及在外来工具、损坏工具和不可解任务下的鲁棒性。在12个MLLM上,APB揭示了长程规划、工具噪声鲁棒性、校准拒绝和推理时改进方面的系统弱点。我们进一步在200个ToolSandbox任务和200个$τ^2$-bench任务上验证了APB,其中APB引导的改进在三个代表性模型上一致提高了计划正确性、计划等级和下游执行指标。因此,APB作为执行基准的上游诊断补充。

英文摘要:

Planning is central to LLM agents: before acting, an agent must decompose goals, select tools, reason over constraints, and decide when a task is infeasible. Yet existing agent evaluations often report only end-to-end success, making it difficult to determine whether failures stem from planning or execution. We introduce Agent Planning Benchmark (APB), a planning-specific diagnostic benchmark with 4,209 multimodal cases across 22 domains and five settings, covering holistic planning, feedback-conditioned step-wise planning, and robustness under extraneous tools, broken tools, and unsolvable tasks. Across 12 MLLMs, APB reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement. We further validate APB on 200 ToolSandbox tasks and 200 $τ^2$-bench tasks, where APB-guided refinement consistently improves plan correctness, plan grade, and downstream execution metrics across three representative models. APB thus serves as an upstream diagnostic complement to execution benchmarks. The APB benchmark and code are available in \href{https://github.com/Mikivishy/AgentPlanningBenchmark}{this URL}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑