We propose a framework for saliency-based, multi-target detection and segmentation of circular-scan, synthetic-aperture-sonar (CSAS) imagery. Our framework relies on a multi-branch, convolutional encoder-decoder network (MB-CEDN). The encoder portion of the MB-CEDN extracts visual contrast features from CSAS images. These features are fed into dual decoders that perform pixel-level segmentation to mask targets. Each decoder provides different perspectives as to what constitutes a salient target. These opinions are aggregated and cascaded into a deep-parsing network to refine the segmentation.
We evaluate our framework using real-world CSAS imagery consisting of five broad target classes. We compare against existing approaches from the computer-vision literature. We show that our framework outperforms supervised, deep-saliency networks designed for natural imagery. It greatly outperforms unsupervised saliency approaches developed for natural imagery. This illustrates that natural-image-based models may need to be altered to be effective for this imaging-sonar modality.
Histogram Layers for Synthetic Aperture Sonar Imagery
Naval Surface Warfare Center Panama City Division（巴拿马城海军水面作战中心分部）；University of Missouri（密苏里大学）；University of Florida（佛罗里达大学）；Texas A&M University（德州农工大学）
Deep learning has not been routinely employed for semantic segmentation of seabed environment for synthetic aperture sonar (SAS) imagery due to the implicit need of abundant training data such methods necessitate. Abundant training data, specifically pixel-level labels for all images, is usually not available for SAS imagery due to the complex logistics (e.g., diver survey, chase boat, precision position information) needed for obtaining accurate ground-truth. Many hand-crafted feature based algorithms have been proposed to segment SAS in an unsupervised fashion. However, there is still room f…
Pennsylvania State University（宾夕法尼亚州立大学）；Centre for Maritime Research & Experimentation（海事研究与实验中心）
Object classification in synthetic aperture sonar (SAS) imagery is usually a data starved and class imbalanced problem. There are few objects of interest present among much benign seafloor. Despite these problems, current classification techniques discard a large portion of the collected SAS information. In particular, a beamformed SAS image, which we call a single-look complex (SLC) image, contains complex pixels composed of real and imaginary parts. For human consumption, the SLC is converted to a magnitude-phase representation and the phase information is discarded. Even more problematic, t…
Observations of seabed texture with synthetic aperture sonar are dependent upon several factors. In this work, we focus on collection geometry with respect to isotropic and anisotropic textures. The low grazing angle of the collection geometry, combined with orientation of the sonar path relative to anisotropic texture, poses a significant challenge for image-alignment and other multi-view scene understanding frameworks. We previously proposed using features captured from estimated seabed relief to improve scene understanding. While several methods have been developed to estimate seabed relief…
Applied Research Labs Penn State（宾夕法尼亚州立大学应用研究实验室）
Performance of automatic target recognition from synthetic aperture sonar data is heavily dependent on the complexity of the beamformed imagery. Several mechanisms can contribute to this, including unwanted vehicle dynamics, the bathymetry of the scene, and the presence of natural and manmade clutter. To understand the impact of the environmental complexity on image perception, researchers have taken approaches rooted in information theory, or heuristics. Despite these efforts, a quantitative measure for complexity has not been related to the phenomenology from which it is derived. By using su…
Synthetic aperture sonar (SAS) requires precise positional and environmental information to produce well-focused output during the image reconstruction step. However, errors in these measurements are commonly present resulting in defocused imagery. To overcome these issues, an \emph{autofocus} algorithm is employed as a post-processing step after image reconstruction for the purpose of improving image quality using the image content itself. These algorithms are usually iterative and metric-based in that they seek to optimize an image sharpness metric. In this letter, we demonstrate the potenti…
Using Frame Theoretic Convolutional Gridding for Robust Synthetic Aperture Sonar Imaging
Department of Electrical Engineering, Pennsylvania State University, State College, PA（电气工程系，宾夕法尼亚州立大学，州立学院，PA）；Department of Mathematics, Dartmouth College, Hanover, NH（数学系，达特茅斯学院，汉诺威，NH）；U.S. Naval Research Laboratory, Washington, DC（美国海军研究实验室，华盛顿，DC）
Synthetic aperture sonar (SAS) requires precise time-of-flight measurements of the transmitted/received waveform to produce well-focused imagery. It is not uncommon for errors in these measurements to be present resulting in image defocusing. To overcome this, an \emph{autofocus} algorithm is employed as a post-processing step after image reconstruction to improve image focus. A particular class of these algorithms can be framed as a sharpness/contrast metric-based optimization. To improve convergence, a hand-crafted weighting function to remove "bad" areas of the image is sometimes applied to…
ROSAR: An Adversarial Re-Training Framework for Robust Side-Scan Sonar Object Detection
RWTH Aachen University（亚琛工业大学）；INESC INOV-Lab（INESC创新实验室）；ISRC (ISEP/P.PORTO)（波尔图高等工程学院/波尔图大学的科学与研究中心）；INESC TEC（INESC技术与科学创新研究所）；Faculdade de Engenharia, Universidade do Porto（波尔图大学工程学院）
Pennsylvania State University（宾夕法尼亚州立大学）；Applied Research Laboratory（应用研究实验室）
Synthetic aperture sonar (SAS) systems produce high-resolution images of the seabed environment. Moreover, deep learning has demonstrated superior ability in finding robust features for automating imagery analysis. However, the success of deep learning is conditioned on having lots of labeled training data, but obtaining generous pixel-level annotations of SAS imagery is often practically infeasible. This challenge has thus far limited the adoption of deep learning methods for SAS segmentation. Algorithms exist to segment SAS imagery in an unsupervised manner, but they lack the benefit of stat…
Coarse-to-Fine Non-Rigid Registration for Side-Scan Sonar Mosaicking
School of Marine Science and Technology, Northwestern Polytechnical University（海洋科学与技术学院，西北工业大学）；Research & Development Institute of Northwestern Polytechnical University（西北工业大学研发院）；Computer Vision and Robotics Research Institute (ViCOROB) of the University of Girona（加泰罗尼亚大学计算机视觉与机器人研究所（ViCOROB））
Subsea images measured by the side scan sonars (SSSs) are necessary visual data in the process of deep-sea exploration by using the autonomous underwater vehicles (AUVs). They could vividly reflect the topography of the seabed, but usually accompanied by complex and severe noise. This paper proposes a deep denoising method for SSS images without high-quality reference data, which uses one single noise SSS image to perform self-supervised denoising. Compared with the classical artificially designed filters, the deep denoising method shows obvious advantages. The denoising experiments are perfor…
Pennsylvania State University（宾夕法尼亚州立大学）；U. S. Naval Research Laboratory（美国海军研究实验室）
Finding mines in Sonar imagery is a significant problem with a great deal of relevance for seafaring military and commercial endeavors. Unfortunately, the lack of enormous Sonar image data sets has prevented automatic target recognition (ATR) algorithms from some of the same advances seen in other computer vision fields. Namely, the boom in convolutional neural nets (CNNs) which have been able to achieve incredible results - even surpassing human actors - has not been an easily feasible route for many practitioners of Sonar ATR. We demonstrate the power of one avenue to incorporating CNNs into…
PhysDNet: Physics-Guided Decomposition Network of Side-Scan Sonar Imagery
School of Marine Science and Technology, Northwestern Polytechnical University（西北工业大学海洋科学与技术学院）；Research & Development Institute of Northwestern Polytechnical University（西北工业大学研发院）；Computer Vision and Robotics Research Institute (ViCOROB) of the University of Girona（加泰罗尼亚理工大学计算机视觉与机器人研究所）
Generating 3D point cloud (PC) data from noisy sonar measurements is a problem that has potential applications for bathymetry mapping, artificial object inspection, mapping of aquatic plants and fauna as well as underwater navigation and localization of vehicles such as submarines. Side-scan sonar sensors are available in inexpensive cost ranges, especially in fish-finders, where the transducers are usually mounted to the bottom of a boat and can approach shallower depths than the ones attached to an Uncrewed Underwater Vehicle (UUV) can. However, extracting 3D information from side-scan sonar…
University of Florida（佛罗里达大学）；University of Missouri（密苏里大学）
Side-look synthetic aperture sonar (SAS) can produce very high quality images of the sea-floor. When viewing this imagery, a human observer can often easily identify various sea-floor textures such as sand ripple, hard-packed sand, sea grass and rock. In this paper, we present the Possibilistic Fuzzy Local Information C-Means (PFLICM) approach to segment SAS imagery into sea-floor regions that exhibit these various natural textures. The proposed PFLICM method incorporates fuzzy and possibilistic clustering methods and leverages (local) spatial information to perform soft segmentation. Results …
Pennsylvania State University（宾夕法尼亚州立大学）；U.S. Naval Research Laboratory（美国海军研究实验室）
Sonar imaging has seen vast improvements over the last few decades due in part to advances in synthetic aperture Sonar (SAS). Sophisticated classification techniques can now be used in Sonar automatic target recognition (ATR) to locate mines and other threatening objects. Among the most promising of these methods is sparse reconstruction-based classification (SRC) which has shown an impressive resiliency to noise, blur, and occlusion. We present a coherent strategy for expanding upon SRC for Sonar ATR that retains SRC's robustness while also being able to handle targets with diverse geometric …
University of Florida（佛罗里达大学）；Cornell University（康奈尔大学）；Naval Surface Warfare Center, Panama City Division（巴拿马城分部海军水面作战中心）；Naval Undersea Warfare Center, Newport（纽波特海军水下作战中心）
We propose a novel method to generate underwater object imagery that is acoustically compliant with that generated by side-scan sonar using the Unreal Engine. We describe the process to develop, tune, and generate imagery to provide representative images for use in training automated target recognition (ATR) and machine learning algorithms. The methods provide visual approximations for acoustic effects such as back-scatter noise and acoustic shadow, while allowing fast rendering with C++ actor in UE for maximizing the size of potential ATR training datasets. Additionally, we provide analysis o…
Synthetic apertures find applications in many fields, such as radar, radio telescopes, microscopy, sonar, ultrasound, LiDAR, and optical imaging. They approximate the signal of a single hypothetical wide aperture sensor with either an array of static small aperture sensors or a single moving small aperture sensor. Common sense in synthetic aperture sampling is that a dense sampling pattern within a wide aperture is required to reconstruct a clear signal. In this article we show that there exists practical limits to both, synthetic aperture size and number of samples for the application of occl…
Deep convolutional neural networks generally perform well in underwater object recognition tasks on both optical and sonar images. Many such methods require hundreds, if not thousands, of images per class to generalize well to unseen examples. However, obtaining and labeling sufficiently large volumes of data can be relatively costly and time-consuming, especially when observing rare objects or performing real-time operations. Few-Shot Learning (FSL) efforts have produced many promising methods to deal with low data availability. However, little attention has been given in the underwater domai…
Applied Research Laboratory（应用研究实验室）；School of EECS at the Pennsylvania State University（宾夕法尼亚州立大学电子工程与计算机科学学院）
Deep learning has been recently shown to improve performance in the domain of synthetic aperture sonar (SAS) image classification. Given the constant resolution with range of a SAS, it is no surprise that deep learning techniques perform so well. Despite deep learning's recent success, there are still compelling open challenges in reducing the high false alarm rate and enabling success when training imagery is limited, which is a practical challenge that distinguishes the SAS classification problem from standard image classification set-ups where training imagery may be abundant. We address th…
Terrain characterisation for online adaptability of automated sonar processing: Lessons learnt from operationally applying ATR to sidescan sonar in MCM applications
MambaDSF: Multi-Scale SSM with Dilated Feature Fusion for Sonar Small Target Detection
School of Information Science and Engineering, Ocean University of China（中国海洋大学信息科学与工程学院）；School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）
University of Florida（佛罗里达大学）；University of Missouri（密苏里大学）
The Possibilistic Fuzzy Local Information C-Means (PFLICM) method is presented as a technique to segment side-look synthetic aperture sonar (SAS) imagery into distinct regions of the sea-floor. In this work, we investigate and present the results of an automated feature selection approach for SAS image segmentation. The chosen features and resulting segmentation from the image will be assessed based on a select quantitative clustering validity criterion and the subset of the features that reach a desired threshold will be used for the segmentation process.
In this paper we present a novel simulation technique for generating high quality images of any predefined resolution. This method can be used to synthesize sonar scans of size equivalent to those collected during a full-length mission, with across track resolutions of any chosen magnitude. In essence, our model extends Generative Adversarial Networks (GANs) based architecture into a conditional recursive setting, that facilitates the continuity of the generated images. The data produced is continuous, realistically-looking, and can also be generated at least two times faster than the real spe…
Applied Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学应用研究实验室）
Advances in unmanned synthetic aperture sonar (SAS) imaging platforms allow for the simultaneous collection of multiband SAS imagery. The imagery is collected over several octaves and the phenomenology's interactions with the sea floor vary greatly over this range -- higher frequencies resolve proud and fine structure of the seafloor while lower frequencies resolve subsurface features and often induce internal resonance in man-made objects. Currently, analysts examine multiband imagery by viewing a single band at a time. This method makes it difficult to ascertain correlations between any pair…
Arizona State University（亚利桑那州立大学）；Pennsylvania State University（宾夕法尼亚州立大学）；NATO STO Centre for Maritime Research & Experimentation（北约科学与技术组织海事研究与实验中心）
Acquisition of Synthetic Aperture Sonar (SAS) datasets is bottlenecked by the costly deployment of SAS imaging systems, and even when data acquisition is possible,the data is often skewed towards containing barren seafloor rather than objects of interest. We present a novel pipeline, called SAS GAN, which couples an optical renderer with a generative adversarial network (GAN) to synthesize realistic SAS images of targets on the seafloor. This coupling enables high levels of SAS image realism while enabling control over image geometry and parameters. We demonstrate qualitative results by presen…
Instituto de Telecomunicações（葡萄牙电信研究所）；Universidade da Beira Interior（贝拉 interior 大学）；University of Tehran（德黑兰大学）；University of Essex（埃塞克斯大学）
Light field imaging is characterized by capturing brightness, color, and directional information of light rays in a scene. This leads to image representations with huge amount of data that require efficient coding schemes. In this paper, lenslet images are rendered into sub-aperture images. These images are organized as a pseudo-sequence input for the HEVC video codec. To better exploit redundancy among the neighboring sub-aperture images and consequently decrease the distances between a sub-aperture image and its references used for prediction, sub-aperture images are divided into four smalle…
University of Florida（佛罗里达大学）；University of Missouri（密苏里大学）
Synthetic Aperture Sonar (SAS) surveys produce imagery with large regions of transition between seabed types. Due to these regions, it is difficult to label and segment the imagery and, furthermore, challenging to score the image segmentations appropriately. While there are many approaches to quantify performance in standard crisp segmentation schemes, drawing hard boundaries in remote sensing imagery where gradients and regions of uncertainty exist is inappropriate. These cases warrant weak labels and an associated appropriate scoring approach. In this paper, a labeling approach and associate…
Taiyuan University of Technology（太原理工大学）；Durham University（杜伦大学）
Wide field small aperture telescopes (WFSATs) are commonly used for fast sky survey. Telescope arrays composed by several WFSATs are capable to scan sky several times per night. Huge amount of data would be obtained by them and these data need to be processed immediately. In this paper, we propose ARGUS (Astronomical taRGets detection framework for Unified telescopes) for real-time transit detection. The ARGUS uses a deep learning based astronomical detection algorithm implemented in embedded devices in each WFSATs to detect astronomical targets. The position and probability of a detection bei…
