Autonomous underwater vehicle (AUV) plays an increasingly important role in ocean exploration. Existing AUVs are usually not fully autonomous and generally limited to pre-planning or pre-programming tasks. Reinforcement learning (RL) and deep reinforcement learning have been introduced into the AUV design and research to improve its autonomy. However, these methods are still difficult to apply directly to the actual AUV system because of the sparse rewards and low learning efficiency. In this paper, we proposed a deep interactive reinforcement learning method for path following of AUV by combining the advantages of deep reinforcement learning and interactive RL. In addition, since the human trainer cannot provide human rewards for AUV when it is running in the ocean and AUV needs to adapt to a changing environment, we further propose a deep reinforcement learning method that learns from both human rewards and environmental rewards at the same time. We test our methods in two path following tasks---straight line and sinusoids curve following of AUV by simulating in the Gazebo platform. Our experimental results show that with our proposed deep interactive RL method, AUV can converge faster than a DQN learner from only environmental reward. Moreover, AUV learning with our deep RL from both human and environmental rewards can also achieve a similar or even better performance than that with the deep interactive RL method and can adapt to the actual environment by further learning from environmental rewards.
Norwegian University of Science and Technology（挪威科技大学）；Oklahoma State University（俄克拉荷马州立大学）
Control theory provides engineers with a multitude of tools to design controllers that manipulate the closed-loop behavior and stability of dynamical systems. These methods rely heavily on insights about the mathematical model governing the physical system. However, in complex systems, such as autonomous underwater vehicles performing the dual objective of path-following and collision avoidance, decision making becomes non-trivial. We propose a solution using state-of-the-art Deep Reinforcement Learning (DRL) techniques, to develop autonomous agents capable of achieving this hybrid objective w…
Universidad Tecnológica del Uruguay (UTEC)（乌拉圭技术大学）；Centro de Ciências Computacionais (C3) of Universidade Federal do Rio Grande (FURG)（里奥格兰德联邦大学计算科学中心）；Universidade Federal de Santa Maria (UFSM)（圣玛丽亚联邦大学）
Previous works showed that Deep-RL can be applied to perform mapless navigation, including the medium transition of Hybrid Unmanned Aerial Underwater Vehicles (HUAUVs). This paper presents new approaches based on the state-of-the-art actor-critic algorithms to address the navigation and medium transition problems for a HUAUV. We show that a double critic Deep-RL with Recurrent Neural Networks improves the navigation performance of HUAUVs using solely range data and relative localization. Our Deep-RL approaches achieved better navigation and transitioning capabilities with a solid generalizatio…
University of Oslo（奥斯陆大学）；Norwegian Defence Research Establishment（挪威国防研究机构）
Reinforcement learning (RL) algorithms are designed to optimize problem-solving by learning actions that maximize rewards, a task that becomes particularly challenging in random and nonstationary environments. Even advanced RL algorithms are often limited in their ability to solve problems in these conditions. In applications such as searching for underwater pollution clouds with autonomous underwater vehicles (AUVs), RL algorithms must navigate reward-sparse environments, where actions frequently result in a zero reward. This paper aims to address these challenges by revisiting and modifying …
Bonn-Rhein-Sieg University of Applied Sciences（波恩-莱茵-锡格应用技术大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
Docking control of an autonomous underwater vehicle (AUV) is a task that is integral to achieving persistent long term autonomy. This work explores the application of state-of-the-art model-free deep reinforcement learning (DRL) approaches to the task of AUV docking in the continuous domain. We provide a detailed formulation of the reward function, utilized to successfully dock the AUV onto a fixed docking platform. A major contribution that distinguishes our work from the previous approaches is the usage of a physics simulator to define and simulate the underwater environment as well as the D…
MBARI（蒙特雷湾水族馆研究所）；Institut de Ciències del Mar, CSIC（西班牙国家研究委员会海洋科学研究所）；Universitat Politècnica de Catalunya, Barcelona Tech.（加泰罗尼亚理工大学（巴塞罗那理工））
Underwater target localization using range-only and single-beacon (ROSB) techniques with autonomous vehicles has been used recently to improve the limitations of more complex methods, such as long baseline and ultra-short baseline systems. Nonetheless, in ROSB target localization methods, the trajectory of the tracking vehicle near the localized target plays an important role in obtaining the best accuracy of the predicted target position. Here, we investigate a Reinforcement Learning (RL) approach to find the optimal path that an autonomous vehicle should follow in order to increase and optim…
Automatic underwater vehicle hull Design optimization is a complex engineering process for generating a UUV hull with optimized properties on a given requirement. First, it involves the integration of involved computationally complex engineering simulation tools. Second, it needs integration of a sample efficient optimization framework with the integrated toolchain. To this end, we integrated the CAD tool called FreeCAD with CFD tool openFoam for automatic design evaluation. For optimization, we chose Bayesian optimization (BO), which is a well-known technique developed for optimizing time-con…
Yale University（耶鲁大学）；Naval Research Laboratory（美国海军研究实验室）；University of Maryland College Park（马里兰大学帕克分校）；Thomas Jefferson High School for Science and Technology（托马斯·杰斐逊科技高中）
Flapping-fin unmanned underwater vehicle (UUV) propulsion systems provide high maneuverability for naval tasks such as surveillance and terrain exploration. Recent work has explored the use of time-series neural network surrogate models to predict thrust from vehicle design and fin kinematics. We develop a search-based inverse model that leverages a kinematics-to-thrust neural network model for control system design. Our inverse model finds a set of fin kinematics with the multi-objective goal of reaching a target thrust and creating a smooth kinematic transition between flapping cycles. We de…
On-board Sonar Data Classification for Path Following in Underwater Vehicles using Fast Interval Type-2 Fuzzy Extreme Learning Machine
Hamlyn Centre for Robotic Surgery, Imperial College London（机器人手术哈明中心，帝国理工学院伦敦分校）；Center for Engineering and Industrial Development (CIDESI)（工程与工业发展中心（CIDESI））
University of Guelph（圭尔夫大学）；McMaster University（麦克马斯特大学）
Tracking control has been a vital research topic in robotics. This paper presents a novel hybrid control strategy for an unmanned underwater vehicle (UUV) based on a bioinspired neural dynamics model. An enhanced backstepping kinematic control strategy is first developed to avoid sharp velocity jumps and provides smooth velocity commands relative to conventional methods. Then, a novel sliding mode control is proposed, which is capable of providing smooth and continuous torque commands free from chattering. In comparative studies, the proposed combined hybrid control strategy has ensured contro…
NATO STO Centre for Maritime Research and Experimentation (CMRE)（北约科技组织海事研究与实验中心）
In this paper we address the mine countermeasures (MCM) search problem for an autonomous underwater vehicle (AUV) surveying the seabed using a side-looking sonar. We propose a coverage path planning method that adapts the AUV track spacing with the objective of collecting better data. We achieve this by shifting the coverage overlap at the tail of the sensor range where the lowest data quality is expected. To assess the algorithm, we collected data from three at-sea experiments. The adaptive survey allowed the AUV to recover from a situation where the sensor range was overestimated and resulte…
In this paper, we consider the problem of caging and eventual capture of an underwater entity using multiple Autonomous Underwater Vehicles (AUVs) in a 3D water volume We solve this problem both with and without taking bathymetry into account. Our proposed algorithm for range-limited sensing in 3D environments captures a finite-speed entity based on sparse and irregular observations. After an isolated initial sighting of the entity, the uncertainty of its whereabouts grows while deployment of the AUV system is underway. To contain the entity, an initial cage, or barrier of sensing footprints, …
An intelligent control strategy is proposed to eliminate the actuator saturation problem that exists in the trajectory tracking process of unmanned underwater vehicles (UUV). The control strategy consists of two parts: for the kinematic modeling part, a fuzzy logic-refined backstepping control is developed to achieve control velocities within acceptable ranges and errors of small fluctuations; on the basis of the velocities deducted by the improved kinematic control, the sliding mode control (SMC) is introduced in the dynamic modeling to obtain corresponding torques and forces that should be a…
Flinders University of South Australia（南澳大利亚弗林德斯大学）
An Autonomous Underwater Vehicle (AUV) should carry out complex tasks in a limited time interval. Since existing AUVs have limited battery capacity and restricted endurance, they should autonomously manage mission time and the resources to perform effective persistent deployment in longer missions. Task assignment requires making decisions subject to resource constraints, while tasks are assigned with costs and/or values that are budgeted in advance. Tasks are distributed in a particular operation zone and mapped by a waypoint covered network. Thus, design an efficient routing-task priority as…
University of Guelph（圭尔夫大学）；China Ship Scientific Research Center（中国船舶科学研究中心）
This paper proposes an intelligent fault-tolerant control (FTC) strategy to tackle the trajectory tracking problem of an underwater vehicle (UV) under thruster damage (power loss) cases and meanwhile resolve the actuator saturation brought by the vehicle's physical constraints. In the proposed control strategy, the trajectory tracking component is formed by a refined backstepping algorithm that controls the velocity variation and a sliding mode control deducts the torque/force outputs; the fault-tolerant component is established based on a Grasshopper Optimization Algorithm (GOA), which provid…
In Autonomous Underwater Vehicles (AUVs) design, hull resistance is an important factor in determining the power requirements and range of vehicle and consequently affect battery size, weight, and volume requirement of the design. In this paper, we leverage on AI-based optimization algorithm along with Computational Fluid Dynamics (CFD) simulation to study the optimal hull design that minimizing the resistance. By running the CFD-based optimization at different operating velocities and turbulence intensity, we want to study/search the possibility of a universal design that will provide least r…
An Autonomous Underwater Vehicle (AUV) needs to acquire a certain degree of autonomy for any particular underwater mission to fulfill the mission objectives successfully and ensure its safety in all stages of the mission in a large scale operating filed. In this paper, a novel combinatorial conflict-free-task assignment strategy consisting an interactive engagement of a local path planner and an adaptive global route planner, is introduced. The method is established upon the heuristic search potency of the Particle Swarm Optimisation (PSO) algorithm to address the discrete nature of routing-ta…
Fraunhofer Institute for Computer Graphics Research (IGD)（弗劳恩霍夫计算机图形研究所）；University of Rostock（罗斯托克大学）
Visual inspection of underwater structures by vehicles, e.g. remotely operated vehicles (ROVs), plays an important role in scientific, military, and commercial sectors. However, the automatic extraction of information using software tools is hindered by the characteristics of water which degrade the quality of captured videos. As a contribution for restoring the color of underwater images, Underwater Denoising Autoencoder (UDAE) model is developed using a denoising autoencoder with U-Net architecture. The proposed network takes into consideration the accuracy and the computation cost to enable…
Autonomous Underwater Vehicles (AUVs) are platforms used for research and exploration of marine environments. However, these types of vehicles face many challenges that hinder their widespread use in the industry. One of the main limitations is obtaining accurate position estimation, due to the lack of GPS signal underwater. This estimation is usually done with Kalman filters. However, new developments in the neuroscience field have shed light on the mechanisms by which mammals are able to obtain a reliable estimation of their current position based on external and internal motion cues. A new …
Scaling Multi Agent Reinforcement Learning for Underwater Acoustic Tracking via Autonomous Vehicles
KEMLG Research Group, Universitat Politècnica de Catalunya Barcelona, Spain（凯姆尔格研究组，巴塞罗那理工大学，西班牙）；Instituto de Ciencias del Mar, Consejo Superior de Investigaciones Científicas, Barcelona, Spain（海洋科学研究所，西班牙国家科学研究委员会，巴塞罗那，西班牙）；KEMLG Research Group, Universitat Politècnica de Catalunya (UPC), and with the HPAI group at Barcelona Supercomputing Center (BSC), Barcelona, Spain（凯姆尔格研究组，巴塞罗那理工大学（UPC），以及巴塞罗那超级计算中心（BSC）的HPAI组，巴塞罗那，西班牙）
Conventional physics-based modeling is a time-consuming bottleneck in control design for complex nonlinear systems like autonomous underwater vehicles (AUVs). In contrast, purely data-driven models, though convenient and quick to obtain, require a large number of observations and lack operational guarantees for safety-critical systems. Data-driven models leveraging available partially characterized dynamics have potential to provide reliable systems models in a typical data-limited scenario for high value complex systems, thereby avoiding months of expensive expert modeling time. In this work …
