The advent of Multimodal Large Language Models, leveraging the power of Large Language Models, has recently demonstrated superior multimodal understanding and reasoning abilities, heralding a new era for artificial general intelligence. However, achieving AGI necessitates more than just comprehension and reasoning. A crucial capability required is effective planning in diverse scenarios, which involves making reasonable decisions based on complex environments to solve real-world problems. Despite its importance, the planning abilities of current MLLMs in varied scenarios remain underexplored. In this paper, we introduce EgoPlan-Bench2, a rigorous and comprehensive benchmark designed to assess the planning capabilities of MLLMs across a wide range of real-world scenarios. EgoPlan-Bench2 encompasses everyday tasks spanning 4 major domains and 24 detailed scenarios, closely aligned with human daily life. EgoPlan-Bench2 is constructed through a semi-automatic process utilizing egocentric videos, complemented by manual verification. Grounded in a first-person perspective, it mirrors the way humans approach problem-solving in everyday life. We evaluate 21 competitive MLLMs and provide an in-depth analysis of their limitations, revealing that they face significant challenges in real-world planning. To further improve the planning proficiency of current MLLMs, we propose a training-free approach using multimodal Chain-of-Thought (CoT) prompting through investigating the effectiveness of various multimodal prompts in complex planning. Our approach enhances the performance of GPT-4V by 10.24 on EgoPlan-Bench2 without additional training. Our work not only sheds light on the current limitations of MLLMs in planning, but also provides insights for future enhancements in this critical area. We have made data and code available at https://qiulu66.github.io/egoplanbench2/.
Faithful-First Reasoning, Planning, and Acting for Multimodal LLMs
Shanghai Jiao Tong University（上海交通大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Australian National University（澳大利亚国立大学）；Fudan University（复旦大学）；University of Sydney（悉尼大学）
Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（认知与决策智能复杂系统重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）
Mind over Space: Can Multimodal Large Language Models Mentally Navigate?
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University（清华大学-博世联合机器学习中心、THBI实验室、BNRist中心、清华大学计算机科学与技术系）；School of Automation Science and Electrical Engineering , Beihang University（北京航空航天大学自动化科学与电气工程学院）；college of AI, Tsinghua University（清华大学人工智能学院）；Department of Computer Science and Technology , Tsinghua University（清华大学计算机科学与技术系）
MMToM-QA: Multimodal Theory of Mind Question Answering
New York University（纽约大学）；Harvard University（哈佛大学）；Massachusetts Institute of Technology（麻省理工学院）；UC San Diego（加州大学圣迭戈分校）；University of Virginia（弗吉尼亚大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（信息多媒体国家重点实验室，计算机学院，北京大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute for Brain and Intelligence, Fudan University（脑与智能研究院，复旦大学）；University of Science and Technology Beijing（北京科技大学）；Beijing Innovation Center of Humanoid Robotics（北京人形机器人创新中心）
Adaptive Informative Path Planning with Multimodal Sensing
Stanford University（斯坦福大学）
自适应信息路径规划（AIPP）问题模型的代理人的任务是在未知的，部分可观察的环境中获取信息的资源约束。AIPP的现有工作主要集中在代表作为代理运动的结果对世界的观察。我们制定了更一般的设置，代理可以选择不同的传感器之间的一些能源的成本，除了穿越环境收集信息。我们称这个问题为AIPPMS（MS for Multimodal Sensing）。AIPPMS需要从能量消耗和信息获取两个方面对感知和运动的影响进行联合推理。我们框架AIPPMS部分可观察马尔可夫决策过程（POMDP），并解决它与在线规划。我们的方法是基于部分可观测蒙特卡罗规划框架的修改，以确保约束的可行性和启发式推出政策量身定制的AIPPMS。我们评估我们的方法在两个领域：一个模拟的搜索和救援方案和一个具有挑战性的扩展经典的RockSample问题。我们发现，我们的方法优于一个经典的AIPP算法，修改AIPPMS，以及使用随机推出政策的在线规划。
University of Maryland, College Park（马里兰大学学院公园分校）；The Ohio State University（俄亥俄州立大学）；Adobe Research（Adobe研究）；State University of New York at Buffalo（纽约州立大学布法罗分校）
