我们提出了LGX（语言引导探索），这是一种面向语言驱动零样本目标导航（L-ZSON）的新型算法，该任务要求具身智能体在从未见过的环境中导航至被独特描述的目标物体。我们的方法利用大语言模型（LLMs）的常识推理能力来做出序列导航决策，同时使用预训练的视觉-语言定位模型实现通用目标检测。我们在RoboTHOR基准上取得了当前最优的零样本目标导航结果，成功率（SR）比现有基线OWL-ViT CLIP on Wheels（OWL CoW）提升超过27%。此外，我们研究了LLM在机器人导航中的应用，分析了多种影响模型输出的提示策略。最后，我们通过真实世界实验展示了该方法的优势，表明LGX在检测并导航至视觉独特物体方面性能更优。
英文摘要：
We present LGX (Language-guided Exploration), a novel algorithm for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON), where an embodied agent navigates to a uniquely described target object in a previously unseen environment. Our approach makes use of Large Language Models (LLMs) for this task by leveraging the LLM's commonsense reasoning capabilities for making sequential navigational decisions. Simultaneously, we perform generalized target object detection using a pre-trained Vision-Language grounding model. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline of the OWL-ViT CLIP on Wheels (OWL CoW). Furthermore, we study the usage of LLMs for robot navigation and present an analysis of various prompting strategies affecting the model output. Finally, we showcase the benefits of our approach via \textit{real-world} experiments that indicate the superior performance of LGX in detecting and navigating to visually unique objects.
In this paper, we focus on a vision-and-language task in which a robot is instructed to execute household tasks. Given an instruction such as "Rinse off a mug and place it in the coffee maker," the robot is required to locate the mug, wash it, and put it in the coffee maker. This is challenging because the robot needs to break down the instruction sentences into subgoals and execute them in the correct order. On the ALFRED benchmark, the performance of state-of-the-art methods is still far lower than that of humans. This is partially because existing methods sometimes fail to infer subgoals th…
EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI
The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Georgia Institute of Technology（佐治亚理工学院）；Meta AI Research（Meta人工智能研究院）
We present a large-scale study of imitating human demonstrations on tasks that require a virtual robot to search for objects in new environments -- (1) ObjectGoal Navigation (e.g. 'find & go to a chair') and (2) Pick&Place (e.g. 'find mug, pick mug, find counter, place mug on counter'). First, we develop a virtual teleoperation data-collection infrastructure -- connecting Habitat simulator running in a web browser to Amazon Mechanical Turk, allowing remote users to teleoperate virtual robots, safely and at scale. We collect 80k demonstrations for ObjectNav and 12k demonstrations for Pick&Place…
Robots excel in performing repetitive and precision-sensitive tasks in controlled environments such as warehouses and factories, but have not been yet extended to embodied AI agents providing assistance in household tasks. Inspired by the catalyzing effect that benchmarks have played in the AI fields such as computer vision and natural language processing, the community is looking for new benchmarks for embodied AI. Prior work in embodied AI benchmark defines tasks using a different formalism, often specific to one environment, simulator or domain, making it hard to develop general and compara…
GuardianBench: A Same-Scene Instruction-Contrastive Benchmark for Latent Contextual Risk in Embodied AI
Shandong University（山东大学）；National University of Singapore（新加坡国立大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Xiaomi Corporation（小米公司）
Creating virtual humans with embodied, human-like perceptual and actuation constraints has the promise to provide an integrated simulation platform for many scientific and engineering applications. We present Dynamic and Autonomous Simulated Human (DASH), an embodied virtual human that, given natural language commands, performs grasp-and-stack tasks in a physically-simulated cluttered environment solely using its own visual perception, proprioception, and touch, without requiring human motion data. By factoring the DASH system into a vision module, a language module, and manipulation modules o…
The Safety Challenge of World Models for Embodied AI Agents: A Review
University of Pisa（比萨大学）；Huawei RAMS Lab（华为RAMS实验室）；Technical University of Munich（慕尼黑技术大学）；Technical University of Berlin（柏林技术大学）；University of Manchester（曼彻斯特大学）；University of Modena and Reggio Emilia（莫德纳和雷吉奥艾米利亚大学）
PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft
Northwestern University（西北大学）；Northeastern University（东北大学）；South China University of Technology（华南理工大学）；Hong Kong Baptist University（香港 Baptist大学）；Beijing Normal - Hong Kong Baptist University（北京师范大学-香港 Baptist大学）
PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulation
Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University, Beijing, 100084, China（计算机科学与技术系，人工智能研究院，清华-博世联合机器学习中心，THBI实验室，BNRist中心，清华大学，北京，100084，中国）；Peng Cheng Laboratory, 518108, China（鹏城实验室，518108，中国）
PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI
State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院（BIGAI））
Generative Multi-Agent Collaboration in Embodied AI: A Systematic Review
Tongji University（同济大学）；East China Normal University（华东师范大学）；Technical University of Munich（慕尼黑工业大学）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海智能自主系统研究院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）
NavTrust: Benchmarking Trustworthiness for Embodied Navigation
Trustworthy Autonomous Systems Laboratory at the University of California, Riverside（加州大学河滨分校可信自主系统实验室）；University of Michigan（密歇根大学）；Workday；University of Southern California（南加州大学）；Texas A&M University（德克萨斯A&M大学）；Lehigh University（莱斯大学）
EnvBridge: Bridging Diverse Environments with Cross-Environment Knowledge Transfer for Embodied AI
Panasonic Connect Co., Ltd.（松下连接有限公司）；National University of Singapore（新加坡国立大学）；Panasonic R&D Center Singapore（松下新加坡研发中心）；Panasonic R&D Center Germany（松下德国研发中心）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Georgia Institute of Technology（佐治亚理工学院）；University of Texas at Dallas（德克萨斯大学达拉斯分校）
CSAIL, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）
We would like robots to achieve purposeful manipulation by placing any instance from a category of objects into a desired set of goal states. Existing manipulation pipelines typically specify the desired configuration as a target 6-DOF pose and rely on explicitly estimating the pose of the manipulated objects. However, representing an object with a parameterized transformation defined on a fixed template cannot capture large intra-category shape variation, and specifying a target pose at a category level can be physically infeasible or fail to accomplish the task -- e.g. knowing the pose and s…
University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）
Benefiting from language flexibility and compositionality, humans naturally intend to use language to command an embodied agent for complex tasks such as navigation and object manipulation. In this work, we aim to fill the blank of the last mile of embodied agents -- object manipulation by following human guidance, e.g., "move the red mug next to the box while keeping it upright." To this end, we introduce an Automatic Manipulation Solver (AMSolver) system and build a Vision-and-Language Manipulation benchmark (VLMbench) based on it, containing various language instructions on categorized robo…
Recent work on audio-visual navigation targets a single static sound in noise-free audio environments and struggles to generalize to unheard sounds. We introduce the novel dynamic audio-visual navigation benchmark in which an embodied AI agent must catch a moving sound source in an unmapped environment in the presence of distractors and noisy sounds. We propose an end-to-end reinforcement learning approach that relies on a multi-modal architecture that fuses the spatial audio-visual information from a binaural audio signal and spatial occupancy maps to encode the features needed to learn a rob…
State Key Laboratory of Robotics and Intelligent Systems（机器人与智能系统国家重点实验室）；Shenyang Institute of Automation（沈阳自动化研究所）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
NEXUS: Continual Learning of Symbolic Constraints for Safe and Robust Embodied Planning
School of Artificial Intelligence and Automation（人工智能与自动化学院）；Huazhong University of Science and Technology（华中科技大学）；School of Software（软件学院）；Henan University（河南大学）；Institute of Information Engineering（信息工程研究所）；Chinese Academy of Sciences（中国科学院）；School of Cyberspace Security（网络空间安全学院）；University of the Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）
Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；AiRiA；Nanjing University of Information Science and Technology（南京信息科学技术大学）
For service robots to become general-purpose in everyday household environments, they need not only a large library of primitive skills, but also the ability to quickly learn novel tasks specified by users. Fine-tuning neural networks on a variety of downstream tasks has been successful in many vision and language domains, but research is still limited on transfer learning between diverse long-horizon tasks. We propose that, compared to reinforcement learning for a new household activity from scratch, home robots can benefit from transferring the value and policy networks trained for similar t…
University of Oxford（牛津大学）；Oxford Robotics Institute（牛津机器人研究所）；Visual Geometry Group（视觉几何组）
Object pose estimation is an important component of most vision pipelines for embodied agents, as well as in 3D vision more generally. In this paper we tackle the problem of estimating the pose of novel object categories in a zero-shot manner. This extends much of the existing literature by removing the need for pose-labelled datasets or category-specific CAD models for training or inference. Specifically, we make the following contributions. First, we formalise the zero-shot, category-level pose estimation problem and frame it in a way that is most applicable to real-world embodied agents. Se…
Facebook AI Research（Facebook 人工智能研究院）；Georgia Institute of Technology（佐治亚理工学院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Simon Fraser University（西蒙弗雷泽大学）；Robotics at Google（谷歌机器人部门）
We revisit the problem of Object-Goal Navigation (ObjectNav). In its simplest form, ObjectNav is defined as the task of navigating to an object, specified by its label, in an unexplored environment. In particular, the agent is initialized at a random location and pose in an environment and asked to find an instance of an object category, e.g., find a chair, by navigating to it. As the community begins to show increased interest in semantic goal specification for navigation tasks, a number of different often-inconsistent interpretations of this task are emerging. This document summarizes the co…
