本文提出 NATURAL PLAN 自然语言规划基准，覆盖旅行、会议和日历排程任务，通过工具输出作为上下文评估 LLM 规划能力，并揭示现有模型在复杂规划场景中的显著不足。
摘要：
我们提出了 NATURAL PLAN，这是一个以自然语言形式呈现的现实规划基准，包含三项关键任务：旅行规划、会议规划和日历排程。我们将评估重点放在 LLM 在拥有完整任务信息时的规划能力上，通过将 Google Flights、Google Maps 和 Google Calendar 等工具的输出作为上下文提供给模型。这消除了在评估 LLM 规划能力时对工具使用环境的需求。我们观察到，NATURAL PLAN 对当前最先进的模型而言是一个具有挑战性的基准。例如，在旅行规划任务中，GPT-4 和 Gemini 1.5 Pro 的解决率分别仅为 31.1% 和 34.8%。我们发现，随着问题复杂度的增加，模型性能会急剧下降：当涉及 10 个城市时，所有模型的表现都低于 5%，这凸显了当前最先进 LLM 在自然语言规划方面存在的显著差距。我们还在 NATURAL PLAN 上进行了广泛的消融研究，以进一步揭示自我修正、少样本泛化以及基于长上下文的上下文内规划等方法在提升 LLM 规划能力方面的有效性与无效性。
英文摘要：
We introduce NATURAL PLAN, a realistic planning benchmark in natural language containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar Scheduling. We focus our evaluation on the planning capabilities of LLMs with full information on the task, by providing outputs from tools such as Google Flights, Google Maps, and Google Calendar as contexts to the models. This eliminates the need for a tool-use environment for evaluating LLMs on Planning. We observe that NATURAL PLAN is a challenging benchmark for state of the art models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only achieve 31.1% and 34.8% solve rate respectively. We find that model performance drops drastically as the complexity of the problem increases: all models perform below 5% when there are 10 cities, highlighting a significant gap in planning in natural language for SoTA LLMs. We also conduct extensive ablation studies on NATURAL PLAN to further shed light on the (in)effectiveness of approaches such as self-correction, few-shot generalization, and in-context planning with long-contexts on improving LLM planning.
Mil-SCORE: Benchmarking Long-Context Geospatial Reasoning and Planning in Large Language Models
University of Maryland, College Park MD, USA（马里兰大学）；ERA Lab, University of South Florida, Tampa FL, USA（佛罗里达大学埃拉实验室）；DEVCOM Army Research Laboratory, Aberdeen Proving Ground MD, USA（国防部陆军研究实验室）；EEHPC Lab, Johns Hopkins University, Baltimore MD, USA（约翰霍普金斯大学EEHPC实验室）
MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Shanghai Jiao Tong University（上海交通大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）
Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Huawei Technologies Co., Ltd（华为技术有限公司）；The University of Hong Kong（香港大学）；Peng Cheng Laboratory, Shenzhen（鹏城实验室）
EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning
Tencent AI Lab（腾讯 AI 实验室）；The University of Hong Kong（香港大学）；ARC Lab, Tencent PCG（腾讯平台与内容事业群 ARC 实验室）；University of California, Berkeley（加州大学伯克利分校）；Peng Cheng Laboratory（鹏城实验室）
Can AI Reason Like an Urban Planner? Benchmarking Large Language Models Against Professional Judgment
School of Architecture and Urban Planning, Shenzhen University（深圳大学建筑与城市规划学院）；Shenzhen Key Laboratory of Urban Spatial Information and Intelligent Modeling（深圳市城市空间信息与智能建模重点实验室）；Department of Urban Planning and Design, The University of Hong Kong（香港大学城市规划与设计系）
Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks
DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning
1 Taobao \& Tmall Group of Alibaba, 2 Institute of Software, Chinese Academy of Science, 3 University of Chinese Academy of Sciences, 4 Renmin University of China, 5 Informatics Department, PUC-Rio
Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Tongji University（同济大学）；Shanghai AI Laboratory（上海人工智能实验室）；Harbin Institute of Technology（哈尔滨工业大学）；Fudan University（复旦大学）；Skywork AI；University of California, Santa Cruz（加州大学圣克鲁兹分校）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）
LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models
Ingenuity Labs Research Institute, Queen’s University（英杰实验室研究 institute，皇后大学）；School of Computing, Queen’s University（计算学院，皇后大学）；Department of Electrical and Computer Engineering, Queen’s University（电气与计算机工程学院，皇后大学）
PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院 Gallagher 学校）；LLM Department, Hunyuan Team, Tencent（腾讯 Hunyuan 团队 LLM 部门）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；The Chinese University of Hong Kong（香港中文大学）
