Robotics at Google（谷歌机器人部门）；TU Berlin（柏林工业大学）；Google Research（谷歌研究院）
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang… 展开作者
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, Pete Florence
Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.
PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignment
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Data Science & Artificial Intelligence Research Institute, China Unicom（中国unicom数据科学与人工智能研究院）；Unicom Data Intelligence, China Unicom（中国unicom数据智能）
ERGeoBench:A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models
Beijing University of Posts and Telecommunications（北京邮电大学）；State Key Laboratory of Networking and Switching Technology（网络与交换技术国家重点实验室）；School of Materials Science and Engineering（材料科学与工程学院）；China Mobile Research Institute（中国移动研究院）；College of Computing and Data Science（计算与数据科学学院）
Humanoid Agent via Embodied Chain-of-Action Reasoning with Multimodal Foundation Models for Zero-Shot Loco-Manipulation
Embodied AI and Robotics (AIR) Lab, New York University, New York, USA and NYUAD Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE（纽约大学Embodied AI和机器人实验室及纽约大学阿布扎赫尔人工智能与机器人中心）；Harvard AI and Robotics Lab, Harvard University, Boston, USA（哈佛大学人工智能与机器人实验室）；Department of Computer Science, University College London, London, UK（伦敦大学学院计算机科学系）；Department of Computer Science, University of Liverpool, UK（利物浦大学计算机科学系）；NYUAD Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE（纽约大学阿布扎赫尔人工智能与机器人中心）
ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
Université de Montréal（蒙特利尔大学）；Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；McMaster University（麦克马斯特大学）；Meta Platforms（Meta平台公司）
LCMF: Lightweight Cross-Modality Mambaformer for Embodied Robotics VQA
School of Software Northwestern Polytechnical University Xi'an, China（软件学院 西安理工大学中国）；Laboratoire L2Tl University Sorbonne Paris Nord Paris, France（L2Tl实验室 索邦巴黎北大学巴黎法国）
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Institute of Trustworthy Embodied AI, Fudan University（复旦大学可信具身人工智能研究院）
Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Midea Group（美的集团）；The Hong Kong University of Science and Technology（香港科技大学）
Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
Northwestern Polytechnical University（西北工业大学）；The University of Georgia（佐治亚大学）；Shaanxi Normal University（陕西师范大学）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）
Evaluating Multimodal Large Language Models with Daily Composite Tasks in Home Environments
State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence（通用人工智能国家重点实验室、北京通用人工智能研究院）；School of Psychological and Cognitive Sciences and Beijing Key Laboratory of Behavior and Mental Health, Key Laboratory of Machine Perception (Ministry of Education), Peking University（心理与认知科学学院及北京行为与心理健康重点实验室、机器感知重点实验室（教育部））；School of Intelligence Science and Technology, Peking University（智能科学与技术学院）
ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
Rice University（里士大学）；U.S. National Institute of Standards and Technology（美国国家标准与技术研究院）；Massachusetts Institute of Technology（麻省理工学院）；Karlsruhe Institute of Technology（卡尔斯鲁厄技术大学）；Autodesk Research（Autodesk研究）；University of California, Berkeley（加州大学伯克利分校）；KTH Royal Institute of Technology（皇家理工学院）；Tsinghua University（清华大学）；Columbia University（哥伦比亚大学）；ASTM International（美国材料与试验协会）；Carnegie Mellon University（卡内基梅隆大学）；German Aerospace Center (DLR)（德国航空航天中心）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；NVIDIA Research（NVIDIA研究）
RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Engineering, Great Bay University（大湾区大学工程学院）；School of Computing and Information Technology, Great Bay University（大湾区大学计算与信息技术学院）
Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments
Stony Brook University（石溪大学）；New York University（纽约大学）；Brown University（布朗大学）；Imperial College London（帝国理工学院）；Middlebury Institute of International Studies at Monterey（明德大学蒙特雷国际研究学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tongji University（同济大学）；Shanghai Qibao Dwight High School（上海七宝德怀特高级中学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Google Research（谷歌研究院）；Seoul National University（首尔国立大学）
Multimodal learning can benefit from the representation power of pretrained Large Language Models (LLMs). However, state-of-the-art transformer based LLMs often ignore negations in natural language and there is no existing benchmark to quantitatively evaluate whether multimodal transformers inherit this weakness. In this study, we present a new multimodal question answering (QA) benchmark adapted from labeled music videos in AudioSet (Gemmeke et al., 2017) with the goal of systematically evaluating if multimodal transformers can perform complex reasoning to recognize new concepts as negation o…
See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm
University College London（伦敦大学学院）；Imperial College London（伦敦帝国学院）；Huawei Noah’s Ark Lab（华为诺亚实验室）；ShanghaiTech University（上海科技大学）；Southern University of Science（南方科技大学）
Embodied Robot Manipulation in the Era of Foundation Models: Planning and Learning Perspectives
Xi’an Jiaotong Univeristy（西安交通大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Chinese Academy of Sciences（中国科学院）；Westlake University（西湖大学）；Zhejiang University（浙江大学）；University of Sydney（悉尼大学）；BAAI（百度人工智能研究院）；Peking University（北京大学）
GenRL: Multimodal-foundation world models for generalization in embodied agents
IDLab, Ghent University（IDLab，根特大学）；VERSES AI Research Lab（VERSES AI研究实验室）；Mila, University of Montreal（米拉研究所，蒙特利尔大学）；ServiceNow Research（ServiceNow研究院）
Facebook AI Research（Facebook人工智能研究）；Georgia Institute of Technology（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tasks, such as semantic goal navigation and embodied question answering. In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks. The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual concepts in the visual represe…
MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents
School of Computer Science, Peking University（北京大学计算机科学学院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Agent AI: Surveying the Horizons of Multimodal Interaction
Microsoft Gaming（微软游戏）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Washington（华盛顿大学）；Microsoft Research, Redmond（微软研究院（雷德蒙德））；Stanford University（斯坦福大学）
University of California, Riverside（加州大学河滨分校）；Mitsubishi Electric Research Labs（三菱电机研究实验室）
Recent years have seen embodied visual navigation advance in two distinct directions: (i) in equipping the AI agent to follow natural language instructions, and (ii) in making the navigable world multimodal, e.g., audio-visual navigation. However, the real world is not only multimodal, but also often complex, and thus in spite of these advances, agents still need to understand the uncertainty in their actions and seek instructions to navigate. To this end, we present AVLEN~ -- an interactive agent for Audio-Visual-Language Embodied Navigation. Similar to audio-visual navigation tasks, the goal…
