Large language models (LLMs) have recently received considerable attention as alternative solutions for task planning. However, comparing the performance of language-oriented task planners becomes difficult, and there exists a dearth of detailed exploration regarding the effects of various factors such as pre-trained model selection and prompt construction. To address this, we propose a benchmark system for automatically quantifying performance of task planning for home-service embodied agents. Task planners are tested on two pairs of datasets and simulators: 1) ALFRED and AI2-THOR, 2) an extension of Watch-And-Help and VirtualHome. Using the proposed benchmark system, we perform extensive experiments with LLMs and prompts, and explore several enhancements of the baseline planner. We expect that the proposed benchmark tool would accelerate the development of language-oriented task planners.
UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following
Shandong University（山东大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Qilu University of Technology（齐鲁工业大学）；Xiaomi Inc（小米公司）；Beijing University Of Technology（北京工业大学）；Hong Kong Polytechnic University（香港理工大学）
SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents
Data Intelligence for Scientific Innovation Lab, Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心科学智能创新数据实验室）；Beihang University（北京航空航天大学）
EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems
School of Software, Harbin Institute of Technology（哈尔滨工业大学软件学院）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；School of Mathematical and Computer Sciences, Heriot-Watt University, Malaysia Campus（赫瑞-瓦特大学马来西亚校区数学与计算机科学学院）；School of Future Science and Engineering, Soochow University（苏州大学未来科学与工程学院）
NormAct: Benchmarking Embodied Agents' Proactive Compliance with Unspoken Social Norms
State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI)（通用人工智能国家重点实验室，北京通用人工智能研究院）；China Academy of Information and Communications Technology（中国信息通信研究院）；ShanghaiTech University（上海科技大学）
Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trends
University of Electronic Science and Technology of China（电子科技大学）；Qiyuan Lab（启元实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）；Tsinghua University（清华大学）；Beihang University（北京航空航天大学）
EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；State Key Laboratory of General Artificial Intelligence, BIGAI（北京通用人工智能研究院通用人工智能国家重点实验室）；University of Hong Kong（香港大学）
EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI
The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
GuardianBench: A Same-Scene Instruction-Contrastive Benchmark for Latent Contextual Risk in Embodied AI
Shandong University（山东大学）；National University of Singapore（新加坡国立大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Xiaomi Corporation（小米公司）
RoboFactory: Exploring Embodied Agent Collaboration with Compositional Constraints
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Oxford（牛津大学）；HKU（香港大学）；Sun Yat-sen University（中山大学）；Tongji University（同济大学）；Shanghai Jiao Tong University（上海交通大学）
Pipette: An Embodied Simulation Platform, Benchmark, and Data-Efficient Augmentation Framework for Wet-Lab Robotics
Key Laboratory of Smart Manufacturing in Energy Chemical Process Ministry of Education（能源化工过程智能制造国家重点实验室）；Department of Computer Science and Engineering（计算机科学与工程系）；Department of Laboratory Medicine（实验室医学系）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）
ConEQsA: Concurrent and Asynchronous Embodied Questions Scheduling and Answering
Software Engineering Institute, East China Normal University（东华大学软件工程学院）；Department of Computer Science, Yale University（耶鲁大学计算机科学系）；School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；College of Connected Computing, Vanderbilt University（范德比大学连接计算学院）
Embodied-BenchClaw: An Autonomous Multi-Agent System for Embodied Spatial Intelligence Benchmark Construction
QiYuan Lab（启元实验室）；School of Information and Software Engineering, University of Electronic Science and Technology of China（电子科技大学信息与软件工程学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction
QiYuan Lab（启元实验室）；University of Electronic Science and Technology of China（电子科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Northeastern University（东北大学）；Beihang University（北京航空航天大学）；Tsinghua University（清华大学）
SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents
Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences（脑启发认知AI实验室，自动化研究所，中国科学院）；Beijing Key Laboratory of Safe AI and Superalignment（北京安全AI与超对齐关键实验室）；Beijing Institute of AI Safety and Governance（北京AI安全与治理研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Long-term AI（长期AI）
