Learning to Act with Affordance-Aware Multimodal Neural SLAM
Amazon Alexa AI（亚马逊Alexa人工智能）；University of Southern California（南加州大学）；University of California, San Diego（加州大学圣迭戈分校）；University of California, Los Angeles（加州大学洛杉矶分校）
Recent years have witnessed an emerging paradigm shift toward embodied artificial intelligence, in which an agent must learn to solve challenging tasks by interacting with its environment. There are several challenges in solving embodied multimodal tasks, including long-horizon planning, vision-and-language grounding, and efficient exploration. We focus on a critical bottleneck, namely the performance of planning and navigation. To tackle this challenge, we propose a Neural SLAM approach that, for the first time, utilizes several modalities for exploration, predicts an affordance-aware semantic map, and plans over it at the same time. This significantly improves exploration efficiency, leads to robust long-horizon planning, and enables effective vision-and-language grounding. With the proposed Affordance-aware Multimodal Neural SLAM (AMSLAM) approach, we obtain more than 40% improvement over prior published work on the ALFRED benchmark and set a new state-of-the-art generalization performance at a success rate of 23.48% on the test unseen scenes.
The University of Adelaide（阿德莱德大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）；NVIDIA（英伟达）；Queensland University of Technology（昆士兰科技大学）
Simultaneous Localization And Mapping (SLAM) is a fundamental problem in mobile robotics. While point-based SLAM methods provide accurate camera localization, the generated maps lack semantic information. On the other hand, state of the art object detection methods provide rich information about entities present in the scene from a single image. This work marries the two and proposes a method for representing generic objects as quadrics which allows object detections to be seamlessly integrated in a SLAM framework. For scene coverage, additional dominant planar structures are modeled as infini…
Massachusetts Institute of Technology（麻省理工学院）；Laboratory for Information and Decision Systems（信息与决策系统实验室）；Computer Science and Artificial Intelligence Laboratory（计算机科学与人工智能实验室）
Simultaneous localization and mapping (SLAM) is the process of constructing a global model of an environment from local observations of it; this is a foundational capability for mobile robots, supporting such core functions as planning, navigation, and control. This article reviews recent progress in SLAM, focusing on advances in the expressive capacity of the environmental models used in SLAM systems (representation) and the performance of the algorithms used to estimate these models from data (inference). A prominent theme of recent SLAM research is the pursuit of environmental representatio…
Deploying autonomous robots capable of exploring unknown environments has long been a topic of great relevance to the robotics community. In this work, we take a further step in that direction by presenting an open-source active visual SLAM framework that leverages the accuracy of a state-of-the-art graph-SLAM system and takes advantage of the fast utility computation that exploiting the structure of the underlying pose-graph offers. Through careful estimation of a posteriori weighted pose-graphs, D-optimal decision-making is achieved online with the objective of improving localization and map…
Laboratoire des Sciences du Numérique de Nantes (LS2N)（南特数字科学实验室（LS2N））；CNRS（法国国家科学研究中心）；Ecole Centrale de Nantes（南特中央理工学院）；Capgemini Engineering（凯捷工程）
Instituto de Investigación en Ingeniería de Aragón (I3A)（阿拉贡工程研究院（I3A））；Universidad de Zaragoza（萨拉戈萨大学）
Quantifying uncertainty is a key stage in active simultaneous localization and mapping (SLAM), as it allows to identify the most informative actions to execute. However, dealing with full covariance or even Fisher information matrices (FIMs) is computationally heavy and easily becomes intractable for online systems. In this work, we study the paradigm of active graph-SLAM formulated over \textit{SE(n)}, and propose a general relationship between the FIM of the system and the Laplacian matrix of the underlying pose-graph. This link makes possible to use graph connectivity indices as utility fun…
Dream-SLAM: Dreaming the Unseen for Active SLAM in Dynamic Environments
Thrust of Robotics and Autonomous Systems, Hong Kong University of Science and Technology (Guangzhou)（机器人与自主系统研究所，香港科学与技术大学（广州））；University of Zaragoza（阿拉瓦大学）；School of Computation, Information and Technology, Technical University of Munich（计算、信息与技术学院，慕尼黑技术大学）；School of Automation and IntelligentSensing, Shanghai Jiao Tong University（自动化与智能感知学院，上海交通大学）
MA-SLAM: Active SLAM in Large-Scale Unknown Environment using Map Aware Deep Reinforcement Learning
Sun Yat-sen University（中山大学）；South China University of Technology（华南理工大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；City University of Hong Kong（香港城市大学）
Active SLAM is the task of actively planning robot paths while simultaneously building a map and localizing within. Existing work has focused on planning paths with occupancy grid maps, which do not scale well and suffer from long term drift. This work proposes a Topological Feature Graph (TFG) representation that scales well and develops an active SLAM algorithm with it. The TFG uses graphical models, which utilize independences between variables, and enables a unified quantification of exploration and exploitation gains with a single entropy metric. Hence, it facilitates a natural and princi…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Stuttgart（斯图加特大学）
In active Visual-SLAM (V-SLAM), a robot relies on the information retrieved by its cameras to control its own movements for autonomous mapping of the environment. Cameras are usually statically linked to the robot's body, limiting the extra degrees of freedom for visual information acquisition. In this work, we overcome the aforementioned problem by introducing and leveraging an independently rotating camera on the robot base. This enables us to continuously control the heading of the camera, obtaining the desired optimal orientation for active V-SLAM, without rotating the robot itself. Howeve…
Massive Parallel Deep Reinforcement Learning for Active SLAM
Facultad de Ciencias Exactas y Naturales, Universidad de Buenos Aires（布宜诺斯艾利斯大学自然科学与精确科学学院）；Instituto Tecnológico de Aragón (ITA)（阿拉贡技术研究所）；University of Zaragoza（萨拉戈萨大学）；INTELYMEC, Centro de Investigaciones en Física e Ingeniería del Centro (CIFICEN), UNICEN-CICPBA-CONICET（INTELYMEC，中部物理与工程研究中心（CIFICEN），国立中部大学-布宜诺斯艾利斯省科学研究委员会-国家科学与技术研究委员会）
Facebook AI Research（脸书人工智能研究院）；Carnegie Mellon University（卡内基梅隆大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）
This work presents a modular and hierarchical approach to learn policies for exploring 3D environments, called `Active Neural SLAM'. Our approach leverages the strengths of both classical and learning-based methods, by using analytical path planners with learned SLAM module, and global and local policies. The use of learning provides flexibility with respect to input modalities (in the SLAM module), leverages structural regularities of the world (in global policies), and provides robustness to errors in state estimation (in local policies). Such use of learning within each module retains its b…
A Novel ViDAR Device With Visual Inertial Encoder Odometry and Reinforcement Learning-Based Active SLAM Method
State Key Laboratory for Turbulence and Complex Systems, Department of Advanced Manufacturing and Robotics, College of Engineering, Peking University（turbulence和复杂系统国家重点实验室，先进制造与机器人系，工程学院，北京大学）；Tencent RoboticsX（腾讯机器人科技）；College of Information and Electrical Engineering, China Agricultural University（信息与电气工程学院，中国农业大学）
This paper proposes a novel active Simultaneous Localization and Mapping (SLAM) method with continuous trajectory optimization over a stochastic robot dynamics model. The problem is formalized as a stochastic optimal control over the continuous robot kinematic model to minimize a cost function that involves the covariance matrix of the landmark states. We tackle the problem by separately obtaining an open-loop control sequence subject to deterministic dynamics by iterative Covariance Regulation (iCR) and a closed-loop feedback control under stochastic robot and covariance dynamics by Linear Qu…
AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
The University of Hong Kong（香港大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；Knowin AI（考拉悠然科技）
Cross-Embodied Affordance Transfer through Learning Affordance Equivalences
Bogazici University（博加齐大学）；the University of Tokyo（东京大学）；Osaka University（大阪大学）；International Professional University of Technology in Osaka（大阪国际职业技术大学）；University of Trento（特伦托大学）；Ozyegin University（厄齐耶因大学）
We propose a novel system for action sequence planning based on a combination of affordance recognition and a neural forward model predicting the effects of affordance execution. By performing affordance recognition on predicted futures, we avoid reliance on explicit affordance effect definitions for multi-step planning. Because the system learns affordance effects from experience data, the system can foresee not just the canonical effects of an affordance, but also situation-specific side-effects. This allows the system to avoid planning failures due to such non-canonical effects, and makes i…
The ability to autonomously explore and navigate a physical space is a fundamental requirement for virtually any mobile autonomous agent, from household robotic vacuums to autonomous vehicles. Traditional SLAM-based approaches for exploration and navigation largely focus on leveraging scene geometry, but fail to model dynamic objects (such as other agents) or semantic constraints (such as wet floors or doorways). Learning-based RL agents are an attractive alternative because they can incorporate both semantic and geometric information, but are notoriously sample inefficient, difficult to gener…
Stanford Vision and Learning Lab（斯坦福视觉与学习实验室）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Planning in realistic environments requires searching in large planning spaces. Affordances are a powerful concept to simplify this search, because they model what actions can be successful in a given situation. However, the classical notion of affordance is not suitable for long horizon planning because it only informs the robot about the immediate outcome of actions instead of what actions are best for achieving a long-term goal. In this paper, we introduce a new affordance representation that enables the robot to reason about the long-term effects of actions through modeling what actions ar…
Humans form mental images of 3D scenes to support counterfactual imagination, planning, and motor control. Our abilities to predict the appearance and affordance of the scene from previously unobserved viewpoints aid us in performing manipulation tasks (e.g., 6-DoF kitting) with a level of ease that is currently out of reach for existing robot learning frameworks. In this work, we aim to build artificial systems that can analogously plan actions on top of imagined images. To this end, we introduce Mental Imagery for Robotic Affordances (MIRA), an action reasoning framework that optimizes actio…
Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions
CFCS, School of CS, PKU（计算机学院，北京大学）；School of EECS, PKU（电子工程系，北京大学）；University of Oxford（牛津大学）；National Key Laboratory for Multimedia Information Processing, School of CS, PKU（多媒体信息处理国家重点实验室，计算机学院，北京大学）
SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy
Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Department of Thoracic Surgery, Peking University People’s Hospital（北京大学人民医院胸外科）；Thoracic Oncology Institute, Peking University People’s Hospital（北京大学人民医院胸部肿瘤研究所）；Research Unit of Intelligence Diagnosis and Treatment in Early Non-small Cell Lung Cancer, Chinese Academy of Medical Sciences（中国医学科学院早期非小细胞肺癌智能诊断与治疗研究组）；Institute of Advanced Clinical Medicine, Peking University（北京大学先进临床医学院）；Beijing Key Laboratory of Innovative Application of Big Data in Lung Cancer, Peking University People’s Hospital（北京大学人民医院肺癌大数据创新应用北京市重点实验室）
RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation
Xiaomi EV（小米电动车）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学 Gallagher人工智能学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）
Laboratoire des Sciences du Numérique de Nantes (LS2N), CNRS（劳伦斯数字科学南特实验室（LS2N），CNRS）；Ecole Centrale de Nantes (ECN)（南特中央理工学院（ECN））；University of Genoa（热那亚大学）
Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Model
The Hong Kong University of Science and Technology (GZ)（香港科技大学（广州））；National University of Singapore（新加坡国立大学）；ShanghaiTech University（上海科技大学）；East China Normal University（华东师范大学）；Nanjing University of Information Science & Technology（南京信息工程大学）；Zhejiang University（浙江大学）；Institute of Automation, Chinese Academy of Science（中国科学院自动化研究所）；Shanghai AI Laboratory（上海人工智能实验室）
Learning Granularity-Aware Affordances from Human-Object Interaction for Tool-Based Functional Dexterous Grasping
School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
