Georgia Tech（佐治亚理工学院）；Facebook AI Research（Facebook人工智能研究院）；Simon Fraser University（西蒙菲莎大学）；Imperial College London（帝国理工学院）；Princeton University（普林斯顿大学）；Intel Labs（英特尔实验室）；UC Berkeley（加州大学伯克利分校）；Google（谷歌）；Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；UC San Diego（加州大学圣地亚哥分校）
Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, Hao Su
We describe a framework for research and evaluation in Embodied AI. Our proposal is based on a canonical task: Rearrangement. A standard task can focus the development of new techniques and serve as a source of trained models that can be transferred to other settings. In the rearrangement task, the goal is to bring a given physical environment into a specified state. The goal state can be specified by object poses, by images, by a description in language, or by letting the agent experience the environment in the goal state. We characterize rearrangement scenarios along different axes and describe metrics for benchmarking rearrangement performance. To facilitate research and exploration, we present experimental testbeds of rearrangement scenarios in four different simulation environments. We anticipate that other datasets will be released and new simulation platforms will be built to support training of rearrangement agents and their deployment on physical systems.
Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）
There has been a significant recent progress in the field of Embodied AI with researchers developing models and algorithms enabling embodied agents to navigate and interact within completely unseen environments. In this paper, we propose a new dataset and baseline models for the task of Rearrangement. We particularly focus on the task of Room Rearrangement: an agent begins by exploring a room and recording objects' initial configurations. We then remove the agent and change the poses and states (e.g., open/closed) of some objects in the room. The agent must restore the initial configurations o…
Georgia Institute of Technology（佐治亚理工学院）；Meta AI；Google Atlanta（谷歌亚特兰大）
We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly different times to generate rollouts) to many GPUs residing on, potentially, many machines. VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods (SyncOnRL and AsyncOnRL, respectively). VER learns from on-policy experience (like SyncOnRL) and has no synchronization points (like AsyncOnRL). VER leads to significant and consistent speed-ups across a bro…
Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）
Massive datasets and high-capacity models have driven many recent advancements in computer vision and natural language understanding. This work presents a platform to enable similar success stories in Embodied AI. We propose ProcTHOR, a framework for procedural generation of Embodied AI environments. ProcTHOR enables us to sample arbitrarily large datasets of diverse, interactive, customizable, and performant virtual environments to train and evaluate embodied agents across navigation, interaction, and manipulation tasks. We demonstrate the power and potential of ProcTHOR via a sample of 10,00…
Carnegie Mellon University（卡内基梅隆大学）；Amazon Alexa AI（亚马逊Alexa人工智能部门）；University of Southern California（南加州大学）
Physically rearranging objects is an important capability for embodied agents. Visual room rearrangement evaluates an agent's ability to rearrange objects in a room to a desired goal based solely on visual input. We propose a simple yet effective method for this problem: (1) search for and map which objects need to be rearranged, and (2) rearrange each object until the task is complete. Our approach consists of an off-the-shelf semantic segmentation model, voxel-based semantic map, and semantic search policy to efficiently find objects that need to be rearranged. On the AI2-THOR Rearrangement …
Columbia University（哥伦比亚大学）；Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）
We propose Continuous Scene Representations (CSR), a scene representation constructed by an embodied agent navigating within a space, where objects and their relationships are modeled by continuous valued embeddings. Our method captures feature relationships between objects, composes them into a graph structure on-the-fly, and situates an embodied agent within the representation. Our key insight is to embed pair-wise relationships between objects in a latent space. This allows for a richer representation compared to discrete relations (e.g., [support], [next-to]) commonly used for building sce…
Allen Institute for AI（艾伦人工智能研究所）；Yonsei University（延世大学）
Embodied AI agents continue to become more capable every year with the advent of new models, environments, and benchmarks, but are still far away from being performant and reliable enough to be deployed in real, user-facing, applications. In this paper, we ask: can we bridge this gap by enabling agents to ask for assistance from an expert such as a human being? To this end, we propose the Ask4Help policy that augments agents with the ability to request, and then use expert assistance. Ask4Help policies can be efficiently trained without modifying the original agent's parameters and learn a des…
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；MIT（麻省理工学院）；Stanford University（斯坦福大学）
We introduce a visually-guided and physics-driven task-and-motion planning benchmark, which we call the ThreeDWorld Transport Challenge. In this challenge, an embodied agent equipped with two 9-DOF articulated arms is spawned randomly in a simulated physical home environment. The agent is required to find a small set of objects scattered around the house, pick them up, and transport them to a desired final location. We also position containers around the house that can be used as tools to assist with transporting objects efficiently. To complete the task, an embodied agent must plan a sequence…
Recent advances in the areas of multimodal machine learning and artificial intelligence (AI) have led to the development of challenging tasks at the intersection of Computer Vision, Natural Language Processing, and Embodied AI. Whereas many approaches and previous survey pursuits have characterised one or two of these dimensions, there has not been a holistic analysis at the center of all three. Moreover, even when combinations of these topics are considered, more focus is placed on describing, e.g., current architectural methods, as opposed to also illustrating high-level challenges and oppor…
Contrastive language image pretraining (CLIP) encoders have been shown to be beneficial for a range of visual tasks from classification and detection to captioning and image manipulation. We investigate the effectiveness of CLIP visual backbones for Embodied AI tasks. We build incredibly simple baselines, named EmbCLIP, with no task specific architectures, inductive biases (such as the use of semantic maps), auxiliary tasks during training, or depth maps -- yet we find that our improved baselines perform very well across a range of tasks and simulators. EmbCLIP tops the RoboTHOR ObjectNav lead…
Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；Georgia Tech（佐治亚理工学院）；Meta AI；Carnegie Mellon University（卡内基梅隆大学）；FBK（布鲁诺·凯塞勒基金会）；University of Padova（帕多瓦大学）；Simon Fraser University（西蒙弗雷泽大学）；UT Austin（得克萨斯大学奥斯汀分校）；NVIDIA（英伟达）；Apple（苹果公司）；Stanford University（斯坦福大学）；Google（谷歌）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；UMass Amherst（马萨诸塞大学阿默斯特分校）；QUT Centre for Robotics（昆士兰科技大学机器人中心）；Oregon State University（俄勒冈州立大学）；Intel Labs（英特尔研究院）
We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.
The Safety Challenge of World Models for Embodied AI Agents: A Review
University of Pisa（比萨大学）；Huawei RAMS Lab（华为RAMS实验室）；Technical University of Munich（慕尼黑技术大学）；Technical University of Berlin（柏林技术大学）；University of Manchester（曼彻斯特大学）；University of Modena and Reggio Emilia（莫德纳和雷吉奥艾米利亚大学）
RoboFactory: Exploring Embodied Agent Collaboration with Compositional Constraints
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Oxford（牛津大学）；HKU（香港大学）；Sun Yat-sen University（中山大学）；Tongji University（同济大学）；Shanghai Jiao Tong University（上海交通大学）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Amazon Alexa AI（亚马逊 Alexa AI）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；University of Southern California（南加利福尼亚大学）
Learning-based methods for training embodied agents typically require a large number of high-quality scenes that contain realistic layouts and support meaningful interactions. However, current simulators for Embodied AI (EAI) challenges only provide simulated indoor scenes with a limited number of layouts. This paper presents Luminous, the first research framework that employs state-of-the-art indoor scene synthesis algorithms to generate large-scale simulated scenes for Embodied AI challenges. Further, we automatically and quantitatively evaluate the quality of generated indoor scenes via the…
Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）
Training embodied agents in simulation has become mainstream for the embodied AI community. However, these agents often struggle when deployed in the physical world due to their inability to generalize to real-world environments. In this paper, we present Phone2Proc, a method that uses a 10-minute phone scan and conditional procedural generation to create a distribution of training scenes that are semantically similar to the target environment. The generated scenes are conditioned on the wall layout and arrangement of large objects from the scan, while also sampling lighting, clutter, surface …
EmbodiedAgent: A Scalable Hierarchical Approach to Overcome Practical Challenge in Multi-Robot Control
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China（香港中文大学（深圳）科学与工程学院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society, China（社会人工智能与机器人研究所）；The School of Computer Science, The University of Sydney, Australia（悉尼大学计算机科学学院）；Faculty of Computer and Mathematical Sciences, The Hong Kong Polytechnic University, China（香港理工大学计算机与数学科学学院）
The Johns Hopkins University（约翰斯·霍普金斯大学）；National University of Singapore（新加坡国立大学）；NVIDIA（英伟达）
Rearrangement tasks have been identified as a crucial challenge for intelligent robotic manipulation, but few methods allow for precise construction of unseen structures. We propose a visual foresight model for pick-and-place rearrangement manipulation which is able to learn efficiently. In addition, we develop a multi-modal action proposal module which builds on the Goal-Conditioned Transporter Network, a state-of-the-art imitation learning method. Our image-based task planning method, Transporters with Visual Foresight, is able to learn from only a handful of data and generalize to multiple …
Monash University（莫纳什大学）；University of Chinese Academy of Sciences（中国科学院大学）；Sun Yat-sen University（中山大学）；RMIT University（皇家墨尔本理工大学）
"Embodied visual navigation" problem requires an agent to navigate in a 3D environment mainly rely on its first-person observation. This problem has attracted rising attention in recent years due to its wide application in autonomous driving, vacuum cleaner, and rescue robot. A navigation agent is supposed to have various intelligent skills, such as visual perceiving, mapping, planning, exploring and reasoning, etc. Building such an agent that observes, thinks, and acts is a key to real intelligence. The remarkable learning ability of deep learning methods empowered the agents to accomplish em…
Massachusetts Institute of Technology（麻省理工学院）；University of Oxford（牛津大学）；University of Toronto（多伦多大学）；Waverly；Mila；University of Montreal（蒙特利尔大学）；Stanford University（斯坦福大学）；Technische Universität Berlin（柏林工业大学）；Element AI – A ServiceNow Company（Element AI – ServiceNow旗下公司）；University of Washington（华盛顿大学）；University of Pennsylvania（宾夕法尼亚大学）；McGill University（麦吉尔大学）；[Google] X
Machine learning has long since become a keystone technology, accelerating science and applications in a broad range of domains. Consequently, the notion of applying learning methods to a particular problem set has become an established and valuable modus operandi to advance a particular field. In this article we argue that such an approach does not straightforwardly extended to robotics -- or to embodied intelligence more generally: systems which engage in a purposeful exchange of energy and information with a physical environment. In particular, the purview of embodied intelligent agents ext…
Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Sun Yat-sen University（中山大学）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）；Peng Cheng Laboratory（鹏城实验室）；Institute of Digital Media, Peking University（北京大学数字媒体研究院）
State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI)（通用人工智能国家重点实验室、北京通用人工智能研究院（BIGAI））；Tsinghua University（清华大学）；Hong Kong University of Science and Technology（香港科技大学）；University of Rochester（罗切斯特大学）
Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）
Visual recognition ecosystems (e.g. ImageNet, Pascal, COCO) have undeniably played a prevailing role in the evolution of modern computer vision. We argue that interactive and embodied visual AI has reached a stage of development similar to visual recognition prior to the advent of these ecosystems. Recently, various synthetic environments have been introduced to facilitate research in embodied AI. Notwithstanding this progress, the crucial question of how well models trained in simulation generalize to reality has remained largely unanswered. The creation of a comparable ecosystem for simulati…
ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning
Huawei Noah’s Ark Lab（华为诺亚实验室）；University of Leeds（利兹大学）；Technical University of Darmstadt（达姆施塔特技术大学）；East China Normal University（华东师范大学）；Huawei Technologies（华为技术有限公司）；ETH Zurich（苏黎世联邦理工学院）；University College London（伦敦大学学院）
Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；AiRiA；Nanjing University of Information Science and Technology（南京信息科学技术大学）
Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
Northwestern Polytechnical University（西北工业大学）；The University of Georgia（佐治亚大学）；Shaanxi Normal University（陕西师范大学）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）
The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models
College of Information Science and Technology, Jinan University（信息科学与技术学院，暨南大学）；Faculty of Data Science, City University of Macau（数据科学学院，澳门城市大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Zhongguancun Laboratory（中关村实验室）；College of Cyber Security, Jinan University（网络安全学院，暨南大学）
Generative Multi-Agent Collaboration in Embodied AI: A Systematic Review
Tongji University（同济大学）；East China Normal University（华东师范大学）；Technical University of Munich（慕尼黑工业大学）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海智能自主系统研究院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室）；University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；Peking University（北京大学）
