Deep convolutional neural networks generally perform well in underwater object recognition tasks on both optical and sonar images. Many such methods require hundreds, if not thousands, of images per class to generalize well to unseen examples. However, obtaining and labeling sufficiently large volumes of data can be relatively costly and time-consuming, especially when observing rare objects or performing real-time operations. Few-Shot Learning (FSL) efforts have produced many promising methods to deal with low data availability. However, little attention has been given in the underwater domain, where the style of images poses additional challenges for object recognition algorithms. To the best of our knowledge, this is the first paper to evaluate and compare several supervised and semi-supervised Few-Shot Learning (FSL) methods using underwater optical and side-scan sonar imagery. Our results show that FSL methods offer a significant advantage over the traditional transfer learning methods that fine-tune pre-trained models. We hope that our work will help apply FSL to autonomous underwater systems and expand their learning capabilities.
This paper proposes a method that combines the style transfer technique and the learned descriptor to enhance the matching performances of underwater sonar images. In the field of underwater vision, sonar is currently the most effective long-distance detection sensor, it has excellent performances in map building and target search tasks. However, the traditional image matching algorithms are all developed based on optical images. In order to solve this contradiction, the style transfer method is used to convert the sonar images into optical styles, and at the same time, the learned descriptor …
Deep Neural Networks have impressive classification performance, but this comes at the expense of significant computational resources at inference time. Autonomous Underwater Vehicles use low-power embedded systems for sonar image perception, and cannot execute large neural networks in real-time. We propose the use of max-pooling aggressively, and we demonstrate it with a Fire-based module and a new Tiny module that includes max-pooling in each module. By stacking them we build networks that achieve the same accuracy as bigger ones, while reducing the number of parameters and considerably incr…
German Research Center for Artificial Intelligence（德国人工智能研究中心）；Bonn-Rhein-Sieg University of Applied Sciences（波恩-莱茵-锡格应用科学大学）；University of Groningen（格罗宁根大学）
Self-supervised learning has proved to be a powerful approach to learn image representations without the need of large labeled datasets. For underwater robotics, it is of great interest to design computer vision algorithms to improve perception capabilities such as sonar image classification. Due to the confidential nature of sonar imaging and the difficulty to interpret sonar images, it is challenging to create public large labeled sonar datasets to train supervised learning algorithms. In this work, we investigate the potential of three self-supervised learning methods (RotNet, Denoising Aut…
In the field of deep-sea exploration, sonar is presently the only efficient long-distance sensing device. The complicated underwater environment, such as noise interference, low target intensity or background dynamics, has brought many negative effects on sonar imaging. Among them, the problem of nonlinear intensity is extremely prevalent. It is also known as the anisotropy of acoustic sensor imaging, that is, when autonomous underwater vehicles (AUVs) carry sonar to detect the same target from different angles, the intensity variation between image pairs is sometimes very large, which makes t…
In this paper, we apply the scattering transform (ST), a nonlinear map based off of a convolutional neural network (CNN), to classification of underwater objects using sonar signals. The ST formalizes the observation that the filters learned by a CNN have wavelet like structure. We achieve effective binary classification both on a real dataset of Unexploded Ordinance (UXOs), as well as synthetically generated examples. We also explore the effects on the waveforms with respect to changes in the object domain (e.g., translation, rotation, and acoustic impedance, etc.), and examine the consequenc…
Nankai University（南开大学）；Tianjin University of Finance and Economics（天津财经大学）；Tsinghua University（清华大学）
We have made a dataset of camouflage object detection mainly for complex seabed scenes, and named it UnderWater RGB&Sonar,or UW-RS for short. The UW-RS dataset contains a total of 1972 image data. The dataset mainly consists of two parts, namely underwater optical data part (UW-R dataset) and underwater sonar data part (UW-S dataset).
Multibeam forward-looking sonar (MFLS) plays an important role in underwater detection. There are several challenges to the research on underwater object detection with MFLS. Firstly, the research is lack of available dataset. Secondly, the sonar image, generally processed at pixel level and transformed to sector representation for the visual habits of human beings, is disadvantageous to the research in artificial intelligence (AI) areas. Towards these challenges, we present a novel dataset, the underwater acoustic target detection (UATD) dataset, consisting of over 9000 MFLS images captured u…
School of Ocean Engineering, Harbin Institute of Technology（哈尔滨工业大学（威海）海洋工程学院）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学（威海）计算机科学与技术学院）
With the continuous development of underwater vision technology, more and more remote sensing images could be obtained. In the underwater scene, sonar sensors are currently the most effective remote perception devices, and the sonar images captured by them could provide rich environment information. In order to analyze a certain scene, we often need to merge the sonar images from different periods, various sonar frequencies and distinctive viewpoints. However, the above scenes will bring nonlinear intensity differences to the sonar images, which will make traditional matching methods almost in…
CIDCO - Development Center for Ocean Mapping（CIDCO——海洋测绘开发中心）
This paper details a new method to recognize and detect underwater objects in real-time sidescan sonar data imagery streams, with case-studies of applications for underwater archeology, and ghost fishing gear retrieval. We first synthesize images from sidescan data, apply geometric and radiometric corrections, then use 2D feature detection algorithms to identify point clouds of descriptive visual microfeatures such as corners and edges in the sonar images. We then apply a clustering algorithm on the feature point clouds to group feature sets into regions of interest, reject false positives, yi…
Sonar-based Deep Learning in Underwater Robotics: Overview, Robustness and Challenges
OceanScan Marine Systems & Technology（OceanScan海洋系统与技术公司）；INESC INOV-Lab（INESC INOV实验室）；ISRC (ISEP/P.PORTO)（ISRC（波尔图高等工程学院/波尔图大学））；Faculty of Engineering, University of Porto (FEUP)（波尔图大学工程学院（FEUP））
Calibration and Comparative Analysis of Forward-Looking Sonar and 3D Sonar for Enhanced Underwater Object Recognition
College of Engineering at the University of Florida（佛罗里达大学工程学院）；Seoul National University（首尔大学）；University of South Florida（南佛罗里达大学）；University of Miami（迈阿密大学）
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Federal University of Bahia (UFBA)（巴伊亚联邦大学（UFBA））
Under the sea, visible spectrum cameras have limited sensing capacity, being able to detect objects only in clear water, but in a constrained range. Considering any sea water condition, sonars are more suitable to support autonomous underwater vehicles' navigation, even in turbid condition. Despite that sonar suitability, this type of sensor does not provide high-density information, such as optical sensors, making the process of object recognition to be more complex. To deal with that problem, we propose a novel trainable method to detect and recognize (identify) specific target objects under…
SonarT165: A Large-scale Benchmark and STFTrack Framework for Acoustic Object Tracking
National Key Laboratory Foundation of Autonomous Marine Vehicle Technology（自主水下机动车辆技术国家重点实验室）；National Natural Science Foundation of China（国家自然科学基金委员会）；National Key Research and Development Program of China（国家重点研发计划）
Exploring the Feasibility of Affordable Sonar Technology: Object Detection in Underwater Environments Using the Ping 360
National Subsea Centre（国家海底中心）；School of Social & Environmental Sustainability, University of Glasgow（格拉斯哥大学社会与环境可持续性学院）；Dataxense（Dataxense公司）；School of Computing, Engineering & Tech., Robert Gordon University（罗伯特戈登大学计算、工程与技术学院）；Bakhtar University（巴克塔大学）
Self-Supervised Enhancement of Forward-Looking Sonar Images: Bridging Cross-Modal Degradation Gaps through Feature Space Transformation and Multi-Frame Fusion
College of Electronic Engineering, Naval University of Engineering（电子工程学院，海军工程大学）；College of Meteorology and Oceanography, National University of Defense Technology（气象海洋学院，国防科技大学）；College of Computer Science and Technology, National University of Defense Technology（计算机科学与技术学院，国防科技大学）；Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）
Hochschule Bonn-Rhein-Sieg（波恩-莱茵-锡格应用技术大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
Application of underwater robots are on the rise, most of them are dependent on sonar for underwater vision, but the lack of strong perception capabilities limits them in this task. An important issue in sonar perception is matching image patches, which can enable other techniques like localization, change detection, and mapping. There is a rich literature for this problem in color images, but for acoustic images, it is lacking, due to the physics that produce these images. In this paper we improve on our previous results for this problem (Valdenegro-Toro et al, 2017), instead of modeling feat…
In this chapter, we present two applications in information fusion in order to evaluate the generalized proportional conflict redistribution rule presented in the chapter \cite{Martin06a}. Most of the time the combination rules are evaluated only on simple examples. We study here different combination rules and compare them in terms of decision on real data. Indeed, in real applications, we need a reliable decision and it is the final results that matter. Two applications are presented here: a fusion of human experts opinions on the kind of underwater sediments depict on sonar image and a clas…
Subsea images measured by the side scan sonars (SSSs) are necessary visual data in the process of deep-sea exploration by using the autonomous underwater vehicles (AUVs). They could vividly reflect the topography of the seabed, but usually accompanied by complex and severe noise. This paper proposes a deep denoising method for SSS images without high-quality reference data, which uses one single noise SSS image to perform self-supervised denoising. Compared with the classical artificially designed filters, the deep denoising method shows obvious advantages. The denoising experiments are perfor…
Garbage and waste disposal is one of the biggest challenges currently faced by mankind. Proper waste disposal and recycling is a must in any sustainable community, and in many coastal areas there is significant water pollution in the form of floating or submerged garbage. This is called marine debris. Submerged marine debris threatens marine life, and for shallow coastal areas, it can also threaten fishing vessels [Iñiguez et al. 2016, Renewable and Sustainable Energy Reviews]. Submerged marine debris typically stays in the environment for a long time (20+ years), and consists of materials tha…
University of Florida（佛罗里达大学）；Cornell University（康奈尔大学）；Naval Surface Warfare Center, Panama City Division（巴拿马城分部海军水面作战中心）；Naval Undersea Warfare Center, Newport（纽波特海军水下作战中心）
We propose a novel method to generate underwater object imagery that is acoustically compliant with that generated by side-scan sonar using the Unreal Engine. We describe the process to develop, tune, and generate imagery to provide representative images for use in training automated target recognition (ATR) and machine learning algorithms. The methods provide visual approximations for acoustic effects such as back-scatter noise and acoustic shadow, while allowing fast rendering with C++ actor in UE for maximizing the size of potential ATR training datasets. Additionally, we provide analysis o…
Matching sonar images with high accuracy has been a problem for a long time, as sonar images are inherently hard to model due to reflections, noise and viewpoint dependence. Autonomous Underwater Vehicles require good sonar image matching capabilities for tasks such as tracking, simultaneous localization and mapping (SLAM) and some cases of object detection/recognition. We propose the use of Convolutional Neural Networks (CNN) to learn a matching function that can be trained from labeled sonar data, after pre-processing to generate matching and non-matching pairs. In a dataset of 39K training …
University of Groningen（格罗宁根大学）；Bonn-Rhein-Sieg University of Applied Sciences（波恩-莱茵-锡格应用科学大学）；Worcester Polytechnic Institute（伍斯特理工学院）；Heriot-Watt University（赫瑞瓦特大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
Generating 3D point cloud (PC) data from noisy sonar measurements is a problem that has potential applications for bathymetry mapping, artificial object inspection, mapping of aquatic plants and fauna as well as underwater navigation and localization of vehicles such as submarines. Side-scan sonar sensors are available in inexpensive cost ranges, especially in fish-finders, where the transducers are usually mounted to the bottom of a boat and can approach shallower depths than the ones attached to an Uncrewed Underwater Vehicle (UUV) can. However, extracting 3D information from side-scan sonar…
The University of Tokyo（东京大学）；JAIST（北陆先端科学技术大学院大学）；Wakachiku Construction Co., Ltd.（若筑建设株式会社）
Retrieving the missing dimension information in acoustic images from 2D forward-looking sonar is a well-known problem in the field of underwater robotics. There are works attempting to retrieve 3D information from a single image which allows the robot to generate 3D maps with fly-through motion. However, owing to the unique image formulation principle, estimating 3D information from a single image faces severe ambiguity problems. Classical methods of multi-view stereo can avoid the ambiguity problems, but may require a large number of viewpoints to generate an accurate model. In this work, we …
Forward-looking sonar can capture high resolution images of underwater scenes, but their interpretation is complex. Generic object detection in such images has not been solved, specially in cases of small and unknown objects. In comparison, detection proposal algorithms have produced top performing object detectors in real-world color images. In this work we develop a Convolutional Neural Network that can reliably score objectness of image windows in forward-looking sonar images and by thresholding objectness, we generate detection proposals. In our dataset of marine garbage objects, we obtain…
In the field of underwater vision research, image matching between the sonar sensors and optical cameras has always been a challenging problem. Due to the difference in the imaging mechanism between them, which are the gray value, texture, contrast, etc. of the acoustic images and the optical images are also variant in local locations, which makes the traditional matching method based on the optical image invalid. Coupled with the difficulties and high costs of underwater data acquisition, it further affects the research process of acousto-optic data fusion technology. In order to maximize the…
FSSUWNet: Mitigating the Fragility of Pre-trained Models with Feature Enhancement for Few-Shot Semantic Segmentation in Underwater Images
School of Ocean Engineering and Technology, Sun Yat-Sen University（中山大学海洋工程学院）；Southern Marine Science and Engineering Guangdong Laboratory（广东海洋科学与工程实验室）
Naval Surface Warfare Center（海军水面作战中心）；Naval Sea Systems Command（海军海上系统司令部）；Naval Undersea Warfare Center（海军水下作战中心）；University of Florida（佛罗里达大学）
We propose a memory-based framework for real-time, data-efficient target analysis in forward-looking-sonar (FLS) imagery. Our framework relies on first removing non-discriminative details from the imagery using a small-scale DenseNet-inspired network. Doing so simplifies ensuing analyses and permits generalizing from few labeled examples. We then cascade the filtered imagery into a novel NeuralRAM-based convolutional matching network, NRMN, for low-shot target recognition. We employ a small-scale FlowNet, LFN to align and register FLS imagery across local temporal scales. LFN enables target la…
I-X and Department of Electrical and Electronic Engineering, Imperial College London, UK（I-X 和 电气与电子工程系，帝国理工学院伦敦分校）；Frontier Robotics, The National Robotarium, Edinburgh UK（前沿机器人技术，国家机器人中心，爱丁堡）
