In this study, we explore the potential of Multimodal Large Language Models (MLLMs) in improving embodied decision-making processes for agents. While Large Language Models (LLMs) have been widely used due to their advanced reasoning skills and vast world knowledge, MLLMs like GPT4-Vision offer enhanced visual understanding and reasoning capabilities. We investigate whether state-of-the-art MLLMs can handle embodied decision-making in an end-to-end manner and whether collaborations between LLMs and MLLMs can enhance decision-making. To address these questions, we introduce a new benchmark called PCA-EVAL, which evaluates embodied decision-making from the perspectives of Perception, Cognition, and Action. Additionally, we propose HOLMES, a multi-agent cooperation framework that allows LLMs to leverage MLLMs and APIs to gather multimodal information for informed decision-making. We compare end-to-end embodied decision-making and HOLMES on our benchmark and find that the GPT4-Vision model demonstrates strong end-to-end embodied decision-making abilities, outperforming GPT4-HOLMES in terms of average decision accuracy (+3%). However, this performance is exclusive to the latest GPT4-Vision model, surpassing the open-source state-of-the-art MLLM by 26%. Our results indicate that powerful MLLMs like GPT4-Vision hold promise for decision-making in embodied agents, offering new avenues for MLLM research. Code and data are open at https://github.com/pkunlp-icler/PCA-EVAL/.
LCMF: Lightweight Cross-Modality Mambaformer for Embodied Robotics VQA
School of Software Northwestern Polytechnical University Xi'an, China（软件学院 西安理工大学中国）；Laboratoire L2Tl University Sorbonne Paris Nord Paris, France（L2Tl实验室 索邦巴黎北大学巴黎法国）
Robotics at Google（谷歌机器人部门）；TU Berlin（柏林工业大学）；Google Research（谷歌研究院）
Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, f…
Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；UAS Esslingen（埃森嫩大学）；TU Wien（维也纳技术大学）；Dr. Ing. h.c. F. Porsche AG（保时捷股份有限公司）；University of Michigan（密歇根大学）；Voxel51 Inc.（Voxel51公司）
Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
Southern University of Science and Technology（南方科技大学）；JD Explore Academy（京东探索研究院）；University of Technology Sydney（悉尼科技大学）；University of Maryland, College Park（马里兰大学学院公园分校）；Yunnan University（云南大学）
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；School of Software, Beihang University（北京航空航天大学软件学院）；Tsinghua University（清华大学）；The University of Sydney（悉尼大学）
Embodied Instruction Following in Unknown Environments
School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications（北京邮电大学智能工程与自动化学院）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；Department of Automation, Tsinghua University（清华大学自动化学院）
Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation
Tsinghua University（清华大学）；Beijing Institute of Technology（北京理工大学）；Beihang University（北航）；State Key Laboratory of General Artificial Intelligence, BIGAI, China（国家一般人工智能重点实验室, BIGAI, 中国）
Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling
University of California Irvine（加州大学欧文分校）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science（保罗·G·艾伦计算机科学学院）
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science and Technology, Beijing Institute of Technology（北京智能信息科技重点实验室，计算机科学与技术学院，北京理工大学）；Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University（广东机器感知与智能计算实验室，深圳MSU-BIT大学）
The Essential Role of Causality in Foundation World Models for Embodied AI
Microsoft Research（微软研究院）；Technical University of Munich（慕尼黑工业大学）；KTH royal institute of technology（皇家理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Reflect After Failure: Brain-Inspired Self-Evolving Agents for Embodied Spatial Intelligence
City University of Hong Kong（香港城市大学）；East China Normal University（华东师范大学）；University of Electronic Science and Technology of China（电子科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The University of Hong Kong（香港大学）
Towards Generalizable Visually Grounded Exploration of Household Devices
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology（哈尔滨工业大学社会计算与交互机器人研究中心）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
Multi-Step Reasoning for Embodied Question Answering via Tool Augmentation
Beijing Key Laboratory of Intelligent Information Technology（北京智能信息科技重点实验室）；Beijing Institute of Technology（北京理工大学）；Guangdong Laboratory of Machine Perception and Intelligent Computing（广东机器感知与智能计算实验室）；Shenzhen MSU-BIT University（深圳MSU-BIT大学）；School of Mechanical Engineering（机械工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（Facebook人工智能研究院）；University of Pennsylvania（宾夕法尼亚大学）
Embodied computer vision considers perception for robots in novel, unstructured environments. Of particular importance is the embodied visual exploration problem: how might a robot equipped with a camera scope out a new environment? Despite the progress thus far, many basic questions pertinent to this problem remain unanswered: (i) What does it mean for an agent to explore its environment well? (ii) Which methods work well, and under which assumptions and environmental settings? (iii) Where do current approaches fall short, and where might future work seek to improve? Seeking answers to these …
EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI
The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
