Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Tongji University(同济大学);Shanghai AI Laboratory(上海人工智能实验室);Harbin Institute of Technology(哈尔滨工业大学);Fudan University(复旦大学);Skywork AI;University of California, Santa Cruz(加州大学圣克鲁兹分校);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学)
Haoyu Sun, Wenxuan Wang, Mingyang Song, Jujie He, Weinan Zhang, Yang Liu, Yang Yang, Yu Cheng
Planning is central to LLM agents: before acting, an agent must decompose goals, select tools, reason over constraints, and decide when a task is infeasible. Yet existing agent evaluations often report only end-to-end success, making it difficult to determine whether failures stem from planning or execution. We introduce Agent Planning Benchmark (APB), a planning-specific diagnostic benchmark with 4,209 multimodal cases across 22 domains and five settings, covering holistic planning, feedback-conditioned step-wise planning, and robustness under extraneous tools, broken tools, and unsolvable tasks. Across 12 MLLMs, APB reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement. We further validate APB on 200 ToolSandbox tasks and 200 $τ^2$-bench tasks, where APB-guided refinement consistently improves plan correctness, plan grade, and downstream execution metrics across three representative models. APB thus serves as an upstream diagnostic complement to execution benchmarks. The APB benchmark and code are available in \href{https://github.com/Mikivishy/AgentPlanningBenchmark}{this URL}.
Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Huawei Technologies Co., Ltd(华为技术有限公司);The University of Hong Kong(香港大学);Peng Cheng Laboratory, Shenzhen(鹏城实验室)
A Super-Learner with Large Language Models for Medical Emergency Advising
Northeastern University(东北大学);Biovitalage S.R.L.(Biovitalage有限责任公司);Territorial Emergency System SET 118(地区紧急系统SET 118);Territorial Center for Medical Assistance(地区医疗援助中心);Aldo Moro University of Bari(巴里阿尔多·莫罗大学)
IMAGINE: Integrating Multi-Agent System into One Model for Complex Reasoning and Planning
Hangzhou International Innovation Institute, Beihang University(北京航空航天大学杭州国际创新研究院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Kuaishou Technology(快手科技)
DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models
Shanghai Jiao Tong University(上海交通大学);SII;SPIRAL Lab(SPIRAL实验室);Generative AI Research Lab (GAIR)(生成式AI研究实验室);Shanghai Chest Hospital(上海胸科医院);Beijing Anzhen Hospital, Capital Medical University(北京安贞医院,首都医科大学)
EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning
Tencent AI Lab(腾讯 AI 实验室);The University of Hong Kong(香港大学);ARC Lab, Tencent PCG(腾讯平台与内容事业群 ARC 实验室);University of California, Berkeley(加州大学伯克利分校);Peng Cheng Laboratory(鹏城实验室)
PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);University of Southern California(南加州大学);Tohoku University(东北大学)
MPO: Boosting LLM Agents with Meta Plan Optimization
Key Laboratory of Computational Linguistics, Ministry of Education, School of Computer Science, Peking University(北京大学计算机学院教育部计算语言学重点实验室);University of Washington(华盛顿大学);Microsoft(微软)