Recent trends in planning research have led to empirical comparison becoming commonplace. The field has started to settle into a methodology for such comparisons, which for obvious practical reasons requires running a subset of planners on a subset of problems. In this paper, we characterize the methodology and examine eight implicit assumptions about the problems, planners and metrics used in many of these comparisons. The problem assumptions are: PR1) the performance of a general purpose planner should not be penalized/biased if executed on a sampling of problems and domains, PR2) minor syntactic differences in representation do not affect performance, and PR3) problems should be solvable by STRIPS capable planners unless they require ADL. The planner assumptions are: PL1) the latest version of a planner is the best one to use, PL2) default parameter settings approximate good performance, and PL3) time cut-offs do not unduly bias outcome. The metrics assumptions are: M1) performance degrades similarly for each planner when run on degraded runtime environments (e.g., machine platform) and M2) the number of plan steps distinguishes performance. We find that most of these assumptions are not supported empirically; in particular, that planners are affected differently by these assumptions. We conclude with a call to the community to devote research resources to improving the state of the practice and especially to enhancing the available benchmark problems.
URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment
Insilico Medicine AI Limited(Insilico Medicine AI有限公司);Independent researcher(独立研究者);Insilico Medicine Canada Inc.(Insilico Medicine加拿大公司);Insilico Medicine Hong Kong Ltd.(Insilico Medicine香港有限公司)
Urban Comfort Assessment in the Era of Digital Planning: A Multidimensional, Data-driven, and AI-assisted Framework
Department of Architecture, National University of Singapore(新加坡国立大学建筑系);School of Engineering and Applied Science, University of Pennsylvania(宾夕法尼亚大学工程与应用科学学院);Department of Real Estate, National University of Singapore(新加坡国立大学房地产系)
COMPARE: Clinical Optimization with Modular Planning and Assessment via RAG-Enhanced AI-OCT: Superior Decision Support for Percutaneous Coronary Intervention Compared to ChatGPT-5 and Junior Operators
Tangdu Hospital, The Fourth Military Medical University(空军军医大学唐都医院);Research Institute of Tsinghua University in Shenzhen(深圳清华大学研究院);The Second Affiliated Hospital, Air Force Medical University(空军军医大学第二附属医院)
A different perspective on a scale for pairwise comparisons
Computer and Automation Research Institute(计算机与自动化研究所);Hungarian Academy of Sciences(匈牙利科学院);Laurentian University(劳伦森大学);Case Western Reserve University(凯斯西储大学);Université Pierre et Marie Curie-Paris 6(皮埃尔和玛丽·居里大学-巴黎第六大学)
EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents
School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);AI Lab, Tencent(腾讯AI实验室);Robotics X, Tencent(腾讯Robotics X实验室);Nanjing University of Posts and Telecommunications(南京邮电大学)
CC-GRMAS: A Multi-Agent Graph Neural System for Spatiotemporal Landslide Risk Assessment in High Mountain Asia
Department of Computer Engineering(计算机工程系);Dwarkadas Jivanlal Sanghvi College of Engineering(德瓦尔卡斯·吉文拉尔·桑格维学院);College of Computing(计算机学院);Georgia Institute of Technology(佐治亚理工学院);Geo-Hydro Meteorological Risks Management Division(地质-水文气象灾害管理部);National Institute of Disaster Management(国家灾害管理研究所)
Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulation
AIRI;Moscow Institute of Physics and Technology (MIPT)(莫斯科物理技术学院);Lomonosov Moscow State University(罗蒙诺索夫莫斯科国立大学);Sberbank, Robotics Center(Sberbank机器人中心);Skoltech
本文提出 Kitchen-R 基准,在 Isaac Sim 数字孪生厨房环境中统一评估任务规划与底层控制,并提供视觉语言模型规划、扩散策略控制及轨迹采集系统,支持独立与集成评估,弥补具身 AI 中语言引导机器人整体评测的空白。
A Comparison of Prompt Engineering Techniques for Task Planning and Execution in Service Robotics
University of Bonn(波恩大学);Lamarr Institute for Machine Learning and Artificial Intelligence(拉马尔机器学习与人工智能研究所);Center for Robotics(机器人中心);Computer Science Institute VI(计算机科学第六研究所);Autonomous Intelligent Systems group(自主智能系统组)
Benchmark for Planning and Control with Large Language Model Agents: Blocksworld with Model Context Protocol
Institute of Automation Technology, Helmut Schmidt University / University of the Federal Armed Forces Hamburg, Germany(自动化技术研究所,海姆·施密特大学/联邦武装部队大学汉堡,德国);Siemens AG, Nuremberg, Germany(西门子股份公司,纽伦堡,德国)