To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task -- Embodied Question Answering [1] in photo-realistic environments (Matterport 3D). We thoroughly study navigation policies that utilize 3D point clouds, RGB images, or their combination. Our analysis of these models reveals several key findings. We find that two seemingly naive navigation baselines, forward-only and random, are strong navigators and challenging to outperform, due to the specific choice of the evaluation setting presented by [1]. We find a novel loss-weighting scheme we call Inflection Weighting to be important when training recurrent models for navigation with behavior cloning and are able to out perform the baselines with this technique. We find that point clouds provide a richer signal than RGB images for learning obstacle avoidance, motivating the use (and continued study) of 3D deep learning models for embodied navigation.
Georgia Institute of Technology（佐治亚理工学院）；Facebook AI Research（Facebook人工智能研究院）
We present a new AI task -- Embodied Question Answering (EmbodiedQA) -- where an agent is spawned at a random location in a 3D environment and asked a question ("What color is the car?"). In order to answer, the agent must first intelligently navigate to explore the environment, gather information through first-person (egocentric) vision, and then answer the question ("orange"). This challenging task requires a range of AI skills -- active perception, language understanding, goal-driven navigation, commonsense reasoning, and grounding of language into actions. In this work, we develop the envi…
University of Cambridge（剑桥大学）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）
Embodied Question Answering (EQA) is a recently proposed task, where an agent is placed in a rich 3D environment and must act based solely on its egocentric input to answer a given question. The desired outcome is that the agent learns to combine capabilities such as scene understanding, navigation and language understanding in order to perform complex reasoning in the visual world. However, initial advancements combining standard vision and language methods with imitation and reinforcement learning algorithms have shown EQA might be too complex and challenging for these techniques. In order t…
We explore blindfold (question-only) baselines for Embodied Question Answering. The EmbodiedQA task requires an agent to answer a question by intelligently navigating in a simulated environment, gathering necessary visual information only through first-person vision before finally answering. Consequently, a blindfold baseline which ignores the environment and visual information is a degenerate solution, yet we show through our experiments on the EQAv1 dataset that a simple question-only baseline achieves state-of-the-art results on the EmbodiedQA task in all cases except when the agent is spaw…
Facebook AI Research（Facebook人工智能研究院）；Georgia Institute of Technology（佐治亚理工学院）
We present a modular approach for learning policies for navigation over long planning horizons from language input. Our hierarchical policy operates at multiple timescales, where the higher-level master policy proposes subgoals to be executed by specialized sub-policies. Our choice of subgoals is compositional and semantic, i.e. they can be sequentially combined in arbitrary orderings, and assume human-interpretable descriptions (e.g. 'exit room', 'find kitchen', 'find refrigerator', etc.). We use imitation learning to warm-start policies at each level of the hierarchy, dramatically increasing…
Facebook AI Research（脸书人工智能研究院）；Simon Fraser University（西蒙弗雷泽大学）；Facebook Reality Labs（脸书现实实验室）；Georgia Institute of Technology（佐治亚理工学院）；Intel Labs（英特尔实验室）；UC Berkeley（加州大学伯克利分校）
We present Habitat, a platform for research in embodied artificial intelligence (AI). Habitat enables training embodied agents (virtual robots) in highly efficient photorealistic 3D simulation. Specifically, Habitat consists of: (i) Habitat-Sim: a flexible, high-performance 3D simulator with configurable agents, sensors, and generic 3D dataset handling. Habitat-Sim is fast -- when rendering a scene from Matterport3D, it achieves several thousand frames per second (fps) running single-threaded, and can reach over 10,000 fps multi-process on a single GPU. (ii) Habitat-API: a modular high-level l…
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science and Technology, Beijing Institute of Technology（北京智能信息科技重点实验室，计算机科学与技术学院，北京理工大学）；Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University（广东机器感知与智能计算实验室，深圳MSU-BIT大学）
IndustryAssetEQA: A Neurosymbolic Operational Intelligence System for Embodied Question Answering in Industrial Asset Maintenance
Artificial Intelligence Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）；University of South Carolina（南卡罗来纳大学）；IBM Yorktown（IBM约克镇分公司）；Indian AI Research Organization（印度人工智能研究组织）
Multi-Step Reasoning for Embodied Question Answering via Tool Augmentation
Beijing Key Laboratory of Intelligent Information Technology（北京智能信息科技重点实验室）；Beijing Institute of Technology（北京理工大学）；Guangdong Laboratory of Machine Perception and Intelligent Computing（广东机器感知与智能计算实验室）；Shenzhen MSU-BIT University（深圳MSU-BIT大学）；School of Mechanical Engineering（机械工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Facebook AI Research（Facebook人工智能研究）；Georgia Institute of Technology（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tasks, such as semantic goal navigation and embodied question answering. In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks. The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual concepts in the visual represe…
RoboTidy : A 3D Gaussian Splatting Household Tidying Benchmark for Embodied Navigation and Action
Huazhong University of Science and Technology（华中科技大学）；The University of Hong Kong（香港大学）；INFIFORCE Intelligent Technology Co., Ltd.（INFIFORCE智能技术有限公司）；Zhejiang University（浙江大学）；Guangming Lab, Shenzhen（深圳光明实验室）
VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Tencent Robotics X（腾讯Robotics X实验室）；University of Toronto（多伦多大学）；Zhili College, Tsinghua University（清华大学志道书院）
Robotics at Google（谷歌机器人部门）；TU Berlin（柏林工业大学）；Google Research（谷歌研究院）
Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, f…
ConEQsA: Concurrent and Asynchronous Embodied Questions Scheduling and Answering
Software Engineering Institute, East China Normal University（东华大学软件工程学院）；Department of Computer Science, Yale University（耶鲁大学计算机科学系）；School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；College of Connected Computing, Vanderbilt University（范德比大学连接计算学院）
E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes
The University of Tokyo, Japan（东京大学）；National Institute of Informatics, Japan（日本信息处理学会）；The University of Osaka, Japan（大阪大学）；Advanced Telecommunications Research Institute International, Japan（国际先进电信研究机构）
