Learning high-level navigation behaviors has important implications: it enables robots to build compact visual memory for repeating demonstrations and to build sparse topological maps for planning in novel environments. Existing approaches only learn discrete, short-horizon behaviors. These standalone behaviors usually assume a discrete action space with simple robot dynamics, thus they cannot capture the intricacy and complexity of real-world trajectories. To this end, we propose Composable Behavior Embedding (CBE), a continuous behavior representation for long-horizon visual navigation. CBE is learned in an end-to-end fashion; it effectively captures path geometry and is robust to unseen obstacles. We show that CBE can be used to performing memory-efficient path following and topological mapping, saving more than an order of magnitude of memory than behavior-less approaches.
AR Forcing: Towards Long-Horizon Robot Navigation World Model
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Shanghai Jiao Tong University（上海交通大学）；School of Safety Science, Tsinghua University（清华大学安全科学学院）；The University of Hong Kong（香港大学）；Lenovo, Beijing, China（北京联想公司）
Learning Traversability for Long Horizon Off-Road Navigation
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University（德克萨斯农工大学J. Mike Walker '66机械工程系）；DEVCOM Army Research laboratory（陆军研究实验室）
School of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）；State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所机器人重点实验室）；State Key Laboratory of Robotics and System, School of Mechanical Engineering and Automation, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）机械工程与自动化学院机器人与系统重点实验室）
In this paper we propose a new framework - MoViLan (Modular Vision and Language) for execution of visually grounded natural language instructions for day to day indoor household tasks. While several data-driven, end-to-end learning frameworks have been proposed for targeted navigation tasks based on the vision and language modalities, performance on recent benchmark data sets revealed the gap in developing comprehensive techniques for long horizon, compositional tasks (involving manipulation and navigation) with diverse object categories, realistic instructions and visual scenarios with non-re…
OnFly: Onboard Zero-Shot Aerial Vision-Language Navigation toward Safety and Efficiency
School of Artificial Intelligence, Sun Yat-Sen University（中山大学人工智能学院）；Southern University of Science and Technology（南方科技大学）；The Hong Kong University of Science and Technology（香港科技大学）
This paper presents a self-supervised Learning from Learned Hallucination (LfLH) method to learn fast and reactive motion planners for ground and aerial robots to navigate through highly constrained environments. The recent Learning from Hallucination (LfH) paradigm for autonomous navigation executes motion plans by random exploration in completely safe obstacle-free spaces, uses hand-crafted hallucination techniques to add imaginary obstacles to the robot's perception, and then learns motion planners to navigate in realistic, highly-constrained, dangerous spaces. However, current hand-crafted…
Polytechnic University of Madrid（马德里理工大学）；Boeing Research & Technology Europe（波音欧洲研究与技术公司）；National University of Distance Education（国立远程教育大学）；Center for Automation and Robotics (UPM-CSIC)（自动化与机器人中心（UPM-CSIC））
This article proposes an inertial navigation algorithm intended to lower the negative consequences of the absence of GNSS (Global Navigation Satellite System) signals on the navigation of autonomous fixed wing low SWaP (Size, Weight, and Power) UAVs (Unmanned Air Vehicles). In addition to accelerometers and gyroscopes, the filter takes advantage of sensors usually present onboard these platforms, such as magnetometers, Pitot tube, and air vanes, and aims to minimize the attitude error and reduce the position drift (both horizontal and vertical) with the dual objective of improving the aircraft…
Shanghai Jiao Tong University（上海交通大学）；Sophia University（上智大学）；the Hong Kong University of Science and Technology（香港科技大学）；Hangzhou ATG Intelligent Technology Co. Ltd.（杭州ATG智能科技有限公司）
In this paper, we present an autonomous unmanned aerial vehicle (UAV) landing system based on visual navigation. We design the landmark as a topological pattern in order to enable the UAV to distinguish the landmark from the environment easily. In addition, a dynamic thresholding method is developed for image binarization to improve detection efficiency. The relative distance in the horizontal plane is calculated according to effective image information, and the relative height is obtained using a linear interpolation method. The landing experiments are performed on a static and a moving platf…
WAM-Nav: Asymmetric Latent World-Action Modeling for Unified Visual Navigation
Nanjing University（南京大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；FiveAges；National University of Defense Technology（国防科技大学）；Tsinghua University（清华大学）；GigaAI（极佳科技）
Centre for Automation and Robotics, Polytechnic University of Madrid（马德里理工大学自动化与机器人中心）
This article proposes a method to diminish the pose (position plus attitude) drift experienced by an SVO (Semi-Direct Visual Odometry) based visual navigation system installed onboard a UAV (Unmanned Air Vehicle) by supplementing its pose estimation non linear optimizations with priors based on the outputs of a GNSS (Global Navigation Satellite System) Denied inertial navigation system. The method is inspired in a PI (Proportional Integral) control system, in which the attitude, altitude, and rate of climb inertial outputs act as targets to ensure that the visual estimations do not deviate far…
We consider the visual feature selection to improve the estimation quality required for the accurate navigation of a robot. We build upon a key property that asserts: contributions of trackable features (landmarks) appear linearly in the information matrix of the corresponding estimation problem. We utilize standard models for motion and vision system using a camera to formulate the feature selection problem over moving finite time horizons. A scalable randomized sampling algorithm is proposed to select more informative features (and ignore the rest) to achieve a superior position estimation q…
MA-CoNav: A Master-Slave Multi-Agent Framework with Hierarchical Collaboration and Dual-Level Reflection for Long-Horizon Embodied VLN
Southwestern University of Finance（西南财经大学）；Department of Informatics,Universitat Hamburg, Hamburg, Germany（汉堡大学信息学院）；University of Electronic Science（电子科学大学）
Humans can robustly follow a visual trajectory defined by a sequence of images (i.e. a video) regardless of substantial changes in the environment or the presence of obstacles. We aim at endowing similar visual navigation capabilities to mobile robots solely equipped with a RGB fisheye camera. We propose a novel probabilistic visual navigation system that learns to follow a sequence of images with bidirectional visual predictions conditioned on possible navigation velocities. By predicting bidirectionally (from start towards goal and vice versa) our method extends its predictive horizon enabli…
Berkeley AI Research (BAIR)（伯克利人工智能研究院（BAIR））；University of California, Berkeley（加州大学伯克利分校）
A general-purpose intelligent robot must be able to learn autonomously and be able to accomplish multiple tasks in order to be deployed in the real world. However, standard reinforcement learning approaches learn separate task-specific policies and assume the reward function for each task is known a priori. We propose a framework that learns event cues from off-policy data, and can flexibly combine these event cues at test time to accomplish different tasks. These event cue labels are not assumed to be known a priori, but are instead labeled using learned models, such as computer vision detect…
Johns Hopkins University Applied Physics Lab（约翰斯·霍普金斯大学应用物理实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）
Safe and high-speed navigation is a key enabling capability for real world deployment of robotic systems. A significant limitation of existing approaches is the computational bottleneck associated with explicit mapping and the limited field of view (FOV) of existing sensor technologies. In this paper, we study algorithmic approaches that allow the robot to predict spaces extending beyond the sensor horizon for robust planning at high speeds. We accomplish this using a generative neural network trained from real-world data without requiring human annotated labels. Further, we extend our existin…
This paper presents a deep learning approach to aid dead-reckoning (DR) navigation using a limited sensor suite. A Recurrent Neural Network (RNN) was developed to predict the relative horizontal velocities of an Autonomous Underwater Vehicle (AUV) using data from an IMU, pressure sensor, and control inputs. The RNN network is trained using experimental data, where a doppler velocity logger (DVL) provided ground truth velocities. The predictions of the relative velocities were implemented in a dead-reckoning algorithm to approximate north and east positions. The studies in this paper were twofo…
University of Texas at Austin（得克萨斯大学奥斯汀分校）；Tokyo Institute of Technology（东京工业大学）
This paper considers the problem of task-dependent (top-down) attention allocation for vision-based autonomous navigation using known landmarks. Unlike the existing paradigm in which landmark selection is formulated as a combinatorial optimization problem, we model it as a resource allocation problem where the decision-maker (DM) is granted extra freedom to control the degree of attention to each landmark. The total resource available to DM is expressed in terms of the capacity limit of the in-take information flow, which is quantified by the directed information from the state of the environm…
Multi-Platform Teach-and-Repeat Navigation by Visual Place Recognition Based on Deep-Learned Local Features
Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（捷克布拉格技术大学信息学、机器人学与控制论研究所）；Faculty of Electrical Engineering, Czech Technical University in Prague（捷克布拉格技术大学电气工程学院）
Facebook AI Research（脸书人工智能研究院）；Carnegie Mellon University（卡内基梅隆大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）
This paper studies the problem of image-goal navigation which involves navigating to the location indicated by a goal image in a novel previously unseen environment. To tackle this problem, we design topological representations for space that effectively leverage semantics and afford approximate geometric reasoning. At the heart of our representations are nodes with associated semantic features, that are interconnected using coarse geometric information. We describe supervised learning-based algorithms that can build, maintain and use such representations under noisy actuation. Experimental st…
Meta AI Research（Meta人工智能研究院）；Carnegie Mellon University（卡内基梅隆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Realistic long-horizon tasks like image-goal navigation involve exploratory and exploitative phases. Assigned with an image of the goal, an embodied agent must explore to discover the goal, i.e., search efficiently using learned priors. Once the goal is discovered, the agent must accurately calibrate the last-mile of navigation to the goal. As with any robust system, switches between exploratory goal discovery and exploitative last-mile navigation enable better recovery from errors. Following these intuitive guide rails, we propose SLING to improve the performance of existing image-goal naviga…
CoReLIN: Constraint-based Reasoning for Zero-shot Lifelong Interactive Navigation
Purdue University（普渡大学）；Indian Institute of Technology, Kharagpur（印度理工学院卡哈拉格普尔分校）；DEVCOM Army Research Lab（DEVCOM陆军研究实验室）；National University of Singapore（新加坡国立大学）
Optimal-Horizon Social Robot Navigation in Heterogeneous Crowds
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；Nation Engineering Research Center for Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究所）；Xi’an Jiaotong University（西安交通大学）
We present a novel approach for efficient and reliable goal-directed long-horizon navigation for a multi-robot team in a structured, unknown environment by predicting statistics of unknown space. Building on recent work in learning-augmented model based planning under uncertainty, we introduce a high-level state and action abstraction that lets us approximate the challenging Dec-POMDP into a tractable stochastic MDP. Our Multi-Robot Learning over Subgoals Planner (MR-LSP) guides agents towards coordinated exploration of regions more likely to reach the unseen goal. We demonstrate improvement i…
