Mapless Navigation of a Hybrid Aerial Underwater Vehicle with Deep Reinforcement Learning Through Environmental Generalization
Universidad Tecnológica del Uruguay (UTEC)（乌拉圭技术大学）；Centro de Ciências Computacionais (C3) of Universidade Federal do Rio Grande (FURG)（里奥格兰德联邦大学计算科学中心）；Universidade Federal de Santa Maria (UFSM)（圣玛丽亚联邦大学）
Ricardo B. Grando, Junior C. de Jesus, Victor A. Kich, Alisson H. Kolling, Rodrigo S. Guerra, Paulo L. J. Drews-Jr
备注：
Paper accepted to the Latin American Robotics Symposium 2022
Previous works showed that Deep-RL can be applied to perform mapless navigation, including the medium transition of Hybrid Unmanned Aerial Underwater Vehicles (HUAUVs). This paper presents new approaches based on the state-of-the-art actor-critic algorithms to address the navigation and medium transition problems for a HUAUV. We show that a double critic Deep-RL with Recurrent Neural Networks improves the navigation performance of HUAUVs using solely range data and relative localization. Our Deep-RL approaches achieved better navigation and transitioning capabilities with a solid generalization of learning through distinct simulated scenarios, outperforming previous approaches.
Universidade Federal do Rio Grande - FURG（里奥格兰德联邦大学（FURG））；Universidade Federal de Santa Maria - UFSM（圣玛丽亚联邦大学（UFSM））；Technological University of Uruguay（乌拉圭技术大学）
Reinforcement Learning (RL) has presented an impressive performance in video games through raw pixel imaging and continuous control tasks. However, RL performs poorly with high-dimensional observations such as raw pixel images. It is generally accepted that physical state-based RL policies such as laser sensor measurements give a more sample-efficient result than learning by pixels. This work presents a new approach that extracts information from a depth map estimation to teach an RL agent to perform the mapless navigation of Unmanned Aerial Vehicle (UAV). We propose the Depth-Imaged Contrasti…
Improving Generalization in Aerial and Terrestrial Mobile Robots Control Through Delayed Policy Learning
Universidad Tecnologica del Uruguay（乌拉圭技术大学）；Universidade Federal de Rio Grande（里奥格兰德联邦大学）；Universidade Federal de Santa Maria（圣玛丽亚联邦大学）；University of Tsukuba（筑波大学）；VersusAI（维瑟斯人工智能公司）
Universidade Federal de Santa Maria - UFSM（圣玛丽亚联邦大学）
Since the application of Deep Q-Learning to the continuous action domain in Atari-like games, Deep Reinforcement Learning (Deep-RL) techniques for motion control have been qualitatively enhanced. Nowadays, modern Deep-RL can be successfully applied to solve a wide range of complex decision-making tasks for many types of vehicles. Based on this context, in this paper, we propose the use of Deep-RL to perform autonomous mapless navigation for Hybrid Unmanned Aerial Underwater Vehicles (HUAUVs), robots that can operate in both, air or water media. We developed two approaches, one deterministic an…
Technological University of Uruguay（乌拉圭技术大学）；Federal University of Rio Grande - FURG（里奥格兰德联邦大学（FURG））；Universidade Federal de Santa Maria - UFSM（圣玛丽亚联邦大学（UFSM））
Deterministic and Stochastic techniques in Deep Reinforcement Learning (Deep-RL) have become a promising solution to improve motion control and the decision-making tasks for a wide variety of robots. Previous works showed that these Deep-RL algorithms can be applied to perform mapless navigation of mobile robots in general. However, they tend to use simple sensing strategies since it has been shown that they perform poorly with a high dimensional state spaces, such as the ones yielded from image-based sensing. This paper presents a comparative analysis of two Deep-RL techniques - Deep Determin…
Technological University of Uruguay（乌拉圭理工大学）；Federal University of Rio Grande（里奥格兰德联邦大学）；Federal University of Santa Maria（圣玛丽亚联邦大学）
This paper presents a novel deep reinforcement learning-based system for 3D mapless navigation for Unmanned Aerial Vehicles (UAVs). Instead of using a image-based sensing approach, we propose a simple learning system that uses only a few sparse range data from a distance sensor to train a learning agent. We based our approaches on two state-of-art double critic Deep-RL models: Twin Delayed Deep Deterministic Policy Gradient (TD3) and Soft Actor-Critic (SAC). We show that our two approaches manage to outperform an approach based on the Deep Deterministic Policy Gradient (DDPG) technique and the…
This study presents a new methodology for learning-based motion planning for autonomous exploration using aerial robots. Through the reinforcement learning method of learning through trial and error, the action policy is derived that can guide autonomous exploration of underground and tunnel environments. A new Markov decision process state is designed to learn the robot's action policy by using simulation only, and the results are applied to the real-world environment without further learning. Reduce the need for the precision map in grid-based path planner and achieve map-less navigation. Th…
Indiana University（印第安纳大学）；University of Southern California（南加利福尼亚大学）
We consider a decision-making problem where the environment varies both in space and time. Such problems arise naturally when considering e.g., the navigation of an underwater robot amidst ocean currents or the navigation of an aerial vehicle in wind. To model such spatiotemporal variation, we extend the standard Markov Decision Process (MDP) to a new framework called the Time-Varying Markov Decision Process (TVMDP). The TVMDP has a time-varying state transition model and transforms the standard MDP that considers only immediate and static uncertainty descriptions of state transitions, to a fr…
SurfAAV: Design and Implementation of a Novel Multimodal Surfing Aquatic-Aerial Vehicle
College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China（智能科学与技术学院，国防科技大学，长沙，中国）；School of Electronic Information, Central South University, Changsha, China（电子信息学院，中南大学，长沙，中国）
University of Electronic Science and Technology of China（电子科技大学）；Dartmouth College（达特茅斯学院）
Underwater navigation presents several challenges, including unstructured unknown environments, lack of reliable localization systems (e.g., GPS), and poor visibility. Furthermore, good-quality obstacle detection sensors for underwater robots are scant and costly; and many sensors like RGB-D cameras and LiDAR only work in-air. To enable reliable mapless underwater navigation despite these challenges, we propose a low-cost end-to-end navigation system, based on a monocular camera and a fixed single-beam echo-sounder, that efficiently navigates an underwater robot to waypoints while avoiding nea…
Univ. Fed. do Rio Grande – FURG（里奥格兰德联邦大学）；Univ. Fed. de Minas Gerais（米纳斯吉拉斯联邦大学）；AuRos Robotics；Universidade de Campinas（坎皮纳斯大学）
In the last decade, a great effort has been employed in the study of Hybrid Unmanned Aerial Underwater Vehicles, robots that can easily fly and dive into the water with different levels of mechanical adaptation. However, most of this literature is concentrated on physical design, practical issues of construction, and, more recently, low-level control strategies. Little has been done in the context of high-level intelligence, such as motion planning and interactions with the real world. Therefore, we proposed in this paper a trajectory planning approach that allows collision avoidance against u…
Bio-inspired vehicles are currently leading the way in the quest to produce a vehicle capable of flight and underwater navigation. However, a fully functional vehicle has not yet been realized. We present the first fully functional vehicle platform operating in air and underwater with seamless transition between both mediums. These unique capabilities combined with the hovering, high maneuverability and reliability of multirotor vehicles, results in a disruptive technology for both civil and military application including air/water search and rescue, inspection, repairs and survey missions amo…
University of Coimbra（科英布拉大学）；Ingeniarius, Ltd.（Ingeniarius有限公司）
Autonomous Underwater Vehicles (AUVs) have the ability to operate in harsh underwater environments without endangering human lives in the process. Nevertheless, just like their ground and aerial counterparts, AUVs need to be able to estimate their own position. Yet, unlike ground and aerial robots, estimating the pose of AUVs is very challenging, with only a few high-cost technological solutions available in the market. In this paper, we present the development of a realistic underwater acoustic model, implemented within the Robot Operating System (ROS) and the Gazebo simulator framework, for …
This paper presents a novel Rapidly-exploring Adaptive Sampling Tree (RAST) algorithm for the adaptive sampling mission of a hybrid aerial underwater vehicle (HAUV) in an air-sea 3D environment. This algorithm innovatively combines the tournament-based point selection sampling strategy, the information heuristic search process and the framework of Rapidly-exploring Random Tree (RRT) algorithm. Hence can guide the vehicle to the region of interest to scientists for sampling and generate a collision-free path for maximizing information collection by the HAUV under the constraints of environmenta…
Parallel Distributional Prioritized Deep Reinforcement Learning for Unmanned Aerial Vehicles
Federal University of Rio Grande（里奥格兰德联邦大学）；University of Tsukuba（筑波大学）；VersusAI；Technological University of Uruguay（乌拉圭理工大学）；Federal University of Santa Maria（圣玛丽亚联邦大学）
Underwater Motions Analysis and Control of a Coupling-Tiltable Unmanned Aerial-Aquatic Vehicle
Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS)（深圳人工智能与机器人研究院）；The Chinese University of Hong Kong（香港中文大学）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
ZS-Puffin: Design, Modeling and Implementation of an Unmanned Aerial-Aquatic Vehicle with Amphibious Wings
School of Ocean Engineering and Technology, Sun Yat‐sen University（海洋工程与技术学院，中山大学）；Southern Marine Science and Engineering Guangdong Laboratory（广东海洋工程科学实验室）；Key Laboratory of Comprehensive Observation of Polar Environment (Sun Yat‐sen University)（极地环境综合观测重点实验室（中山大学））；Guangdong Provincial Key Laboratory of Information Technology for Deep Water Acoustics（广东省深水声学信息技术重点实验室）
Institute for Ocean Technology, National Research Council Canada（加拿大国家研究委员会海洋技术研究所）
This paper presents a mission system and the therein implemented algorithms for path planning in a time-varying environment based on graph methods. The basic task of the introduced path planning algorithms is to find a time-optimal path from a defined start position to a goal position with consideration of the time-varying ocean current for an autonomous underwater vehicle (AUV). Building on this, additional practice-oriented considerations in planning are discussed in this paper. Such points are the discussion of possible methods to accelerate the algorithms and the determination of the optim…
This paper discusses opportunities to parallelize graph based path planning algorithms in a time varying environment. Parallel architectures have become commonplace, requiring algorithm to be parallelized for efficient execution. An additional focal point of this paper is the inclusion of inaccuracies in path planning as a result of forecast error variance, accuracy of calculation in the cost functions and a different observed vehicle speed in the real mission than planned. In this context, robust path planning algorithms will be described. These algorithms are equally applicable to land based…
Massachusetts Institute of Technology (MIT)（麻省理工学院）；Woods Hole Oceanographic Institution (WHOI)（伍兹霍尔海洋研究所）
Successful applications of complex vision-based behaviours underwater have lagged behind progress in terrestrial and aerial domains. This is largely due to the degraded image quality resulting from the physical phenomena involved in underwater image formation. Spectrally-selective light attenuation drains some colors from underwater images while backscattering adds others, making it challenging to perform vision-based tasks underwater. State-of-the-art methods for underwater color correction optimize the parameters of image formation models to restore the full spectrum of color to underwater i…
SHRUMS: Sensor Hallucination for Real-time Underwater Motion Planning with a Compact 3D Sonar
Department of Computer Science and Engineering, Texas A&M University（计算机科学与工程系，德克萨斯A&M大学）；Aquaculture Robotics and Automation, SINTEF Ocean（水产养殖机器人与自动化，SINTEF海洋）
This paper addresses the problem of robotic operations in the presence of adversarial forces. We presents a complete framework for survey operations: waypoint generation,modelling of forces and tuning the control. In many applications of environmental monitoring, search and exploration, and bathymetric mapping, the vehicle has to traverse in straight lines parallel to each other, ensuring there are no gaps and no redundant coverage. During operations with an Autonomous Surface Vehicle (ASV) however, the presence of wind and/or currents produces external forces acting on the vehicle which quite…
