In this paper, we introduce and tackle the simultaneous enhancement and super-resolution (SESR) problem for underwater robot vision and provide an efficient solution for near real-time applications. We present Deep SESR, a residual-in-residual network-based generative model that can learn to restore perceptual image qualities at 2x, 3x, or 4x higher spatial resolution. We supervise its training by formulating a multi-modal objective function that addresses the chrominance-specific underwater color degradation, lack of image sharpness, and loss in high-level feature representation. It is also supervised to learn salient foreground regions in the image, which in turn guides the network to learn global contrast enhancement. We design an end-to-end training pipeline to jointly learn the saliency prediction and SESR on a shared hierarchical feature space for fast inference. Moreover, we present UFO-120, the first dataset to facilitate large-scale SESR learning; it contains over 1500 training samples and a benchmark test set of 120 samples. By thorough experimental evaluation on the UFO-120 and other standard datasets, we demonstrate that Deep SESR outperforms the existing solutions for underwater image enhancement and super-resolution. We also validate its generalization performance on several test cases that include underwater images with diverse spectral and spatial degradation levels, and also terrestrial images with unseen natural objects. Lastly, we analyze its computational feasibility for single-board deployments and demonstrate its operational benefits for visually-guided underwater robots. The model and dataset information will be available at: https://github.com/xahidbuffon/Deep-SESR.
Autonomous underwater vehicles (AUVs) rely on a variety of sensors - acoustic, inertial and visual - for intelligent decision making. Due to its non-intrusive, passive nature, and high information content, vision is an attractive sensing modality, particularly at shallower depths. However, factors such as light refraction and absorption, suspended particles in the water, and color distortion affect the quality of visual data, resulting in noisy and distorted images. AUVs that rely on visual sensing thus face difficult challenges, and consequently exhibit poor performance on vision-driven tasks…
Underwater visual perception is essentially important for underwater exploration, archeology, ecosystem and so on. The low illumination, light reflections, scattering, absorption and suspended particles inevitably lead to the critically degraded underwater image quality, which causes great challenges on recognizing the objects from the underwater images. The existing underwater enhancement methods that aim to promote the underwater visibility, heavily suffer from the poor image restoration performance and generalization ability. To reduce the difficulty of underwater image enhancement, we intr…
In this paper, we present a conditional generative adversarial network-based model for real-time underwater image enhancement. To supervise the adversarial training, we formulate an objective function that evaluates the perceptual image quality based on its global content, color, local texture, and style information. We also present EUVP, a large-scale dataset of a paired and unpaired collection of underwater images (of `poor' and `good' quality) that are captured using seven different cameras over various visibility conditions during oceanic explorations and human-robot collaborative experime…
Ocean University of China（中国海洋大学）；Qingdao University（青岛大学）；China University of Geosciences（中国地质大学）
Due to the wavelength-dependent light attenuation, refraction and scattering, underwater images usually suffer from color distortion and blurred details. However, due to the limited number of paired underwater images with undistorted images as reference, training deep enhancement models for diverse degradation types is quite difficult. To boost the performance of data-driven approaches, it is essential to establish more effective learning mechanisms that mine richer supervised information from limited training sample resources. In this paper, we propose a novel underwater image enhancement net…
CARNet: Collaborative Adversarial Resilience for Robust Underwater Image Enhancement and Perception
School of Software Technology, Dalian University of Technology（大连理工大学软件学院）；College of Information Science and Technology, Dalian Maritime University（大连海事大学信息科学技术学院）；School of Mechanical Engineering, Dalian University of Technology（大连理工大学机械工程学院）
Underwater Variable Zoom: Depth-Guided Perception Network for Underwater Image Enhancement
School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；University of New South Wales（新南威尔士大学）；School of Computer Science and Technology, Shandong Technology and Business University（山东工商学院计算机科学与技术学院）；School of Information and Communication Engineering, Dalian Minzu University（大连民族大学信息与通信工程学院）；School of Innovation Experiment, Dalian University of Technology（大连理工大学创新实验学院）
PIGUIQA: A Physical Imaging Guided Perceptual Framework for Underwater Image Quality Assessment
Chongqing Research Institute of Harbin Institute of Technology（哈尔滨工业大学重庆研究所）；Harbin Institute of Technology（哈尔滨工业大学）；Faculty of Computing（计算机学院）；School of Computer Science（计算机科学学院）；University of Macau（澳门大学）；Institute for Infocomm Research（信息与通信研究所以）
We present a deep residual network-based generative model for single image super-resolution (SISR) of underwater imagery for use by autonomous underwater robots. We also provide an adversarial training pipeline for learning SISR from paired data. In order to supervise the training, we formulate an objective function that evaluates the \textit{perceptual quality} of an image based on its global content, color, and local style information. Additionally, we present USR-248, a large-scale dataset of three sets of underwater images of 'high' (640x480) and 'low' (80x60, 160x120, and 320x240) spatial…
Massachusetts Institute of Technology (MIT)（麻省理工学院）；Woods Hole Oceanographic Institution (WHOI)（伍兹霍尔海洋研究所）
Successful applications of complex vision-based behaviours underwater have lagged behind progress in terrestrial and aerial domains. This is largely due to the degraded image quality resulting from the physical phenomena involved in underwater image formation. Spectrally-selective light attenuation drains some colors from underwater images while backscattering adds others, making it challenging to perform vision-based tasks underwater. State-of-the-art methods for underwater color correction optimize the parameters of image formation models to restore the full spectrum of color to underwater i…
Qingdao University（青岛大学）；Ocean University of China（中国海洋大学）
Achieving subjective and objective quality assessment of underwater images is of high significance in underwater visual perception and image/video processing. However, the development of underwater image quality assessment (UIQA) is limited for the lack of comprehensive human subjective user study with publicly available dataset and reliable objective UIQA metric. To address this issue, we establish a large-scale underwater image dataset, dubbed UID2021, for evaluating no-reference UIQA metrics. The constructed dataset contains 60 multiply degraded underwater images collected from various sour…
This paper reports on WaterGAN, a generative adversarial network (GAN) for generating realistic underwater images from in-air image and depth pairings in an unsupervised pipeline used for color correction of monocular underwater images. Cameras onboard autonomous and remotely operated vehicles can capture high resolution images to map the seafloor, however, underwater image formation is subject to the complex process of light propagation through the water column. The raw images retrieved are characteristically different than images taken in air due to effects such as absorption and scattering,…
Underwater Image Restoration via Polymorphic Large Kernel CNNs
University of Macau（澳门大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen Polytechnic University（深圳职业技术大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Fuzhou University（福州大学）；Peng Cheng Laboratory（鹏城实验室）；Beijing University of Technology（北京工业大学）；Beijing Artificial Intelligence Institute（北京人工智能研究院）；Université de Nantes（南特大学）；Laboratoire des Sciences du Numérique de Nantes（南特数字科学实验室）
The widespread image applications have greatly promoted the vision-based tasks, in which the Image Quality Assessment (IQA) technique has become an increasingly significant issue. For user enjoyment in multimedia systems, the IQA exploits image fidelity and aesthetics to characterize user experience; while for other tasks such as popular object recognition, there exists a low correlation between utilities and perceptions. In such cases, the fidelity-based and aesthetics-based IQA methods cannot be directly applied. To address this issue, this paper proposes a utility-oriented IQA in object rec…
DACA-Net: A Degradation-Aware Conditional Diffusion Network for Underwater Image Enhancement
The Hong Kong University of Science and Technology (Guangzhou), Southern Marine Science and Engineering Guangdong Laboratory (Guangzhou)（香港科学与技术大学（广州）、广东海洋工程科学实验室（广州））；The Hong Kong University of Science（香港科学与技术大学）；Engineering Guangdong Laboratory (Guangzhou)（广东工程实验室（广州））
CIDCO - Development Center for Ocean Mapping（CIDCO——海洋测绘开发中心）
This paper details a new method to recognize and detect underwater objects in real-time sidescan sonar data imagery streams, with case-studies of applications for underwater archeology, and ghost fishing gear retrieval. We first synthesize images from sidescan data, apply geometric and radiometric corrections, then use 2D feature detection algorithms to identify point clouds of descriptive visual microfeatures such as corners and edges in the sonar images. We then apply a clustering algorithm on the feature point clouds to group feature sets into regions of interest, reject false positives, yi…
Sonar-based Deep Learning in Underwater Robotics: Overview, Robustness and Challenges
OceanScan Marine Systems & Technology（OceanScan海洋系统与技术公司）；INESC INOV-Lab（INESC INOV实验室）；ISRC (ISEP/P.PORTO)（ISRC（波尔图高等工程学院/波尔图大学））；Faculty of Engineering, University of Porto (FEUP)（波尔图大学工程学院（FEUP））
In this paper, we present the first large-scale dataset for semantic Segmentation of Underwater IMagery (SUIM). It contains over 1500 images with pixel annotations for eight object categories: fish (vertebrates), reefs (invertebrates), aquatic plants, wrecks/ruins, human divers, robots, and sea-floor. The images have been rigorously collected during oceanic explorations and human-robot collaborative experiments, and annotated by human participants. We also present a benchmark evaluation of state-of-the-art semantic segmentation approaches based on standard performance metrics. In addition, we …
University of Maryland Institute for Advanced Computer Studies（马里兰大学高级计算机研究所）
Oysters are the living vacuum cleaners of the oceans. There is an exponential decline in the oyster population due to over-harvesting. With the current development of the automation and AI, robots are becoming an integral part of the environmental monitoring process that can be also utilized for oyster reef preservation. Nevertheless, the underwater environment poses many difficulties, both from the practical - dangerous and time consuming operations, and the technical perspectives - distorted perception and unreliable navigation. To this end, we present a simulated environment that can be use…
Underwater automatic target recognition (UATR) has been a challenging research topic in ocean engineering. Although deep learning brings opportunities for target recognition on land and in the air, underwater target recognition techniques based on deep learning have lagged due to sensor performance and the size of trainable data. This letter proposed a framework for learning the visual representation of underwater acoustic imageries, which takes a transformer-based style transfer model as the main body. It could replace the low-level texture features of optical images with the visual features …
Mapping the seafloor with underwater imaging cameras is of significant importance for various applications including marine engineering, geology, geomorphology, archaeology and biology. For shallow waters, among the underwater imaging challenges, caustics i.e., the complex physical phenomena resulting from the projection of light rays being refracted by the wavy surface, is likely the most crucial one. Caustics is the main factor during underwater imaging campaigns that massively degrade image quality and affect severely any 2D mosaicking or 3D reconstruction of the seabed. In this work, we pr…
Beneath the Surface: The Role of Underwater Image Enhancement in Object Detection
College of Computing, Michigan Technological University（密歇根理工大学计算学院）；Fairfield University（费尔菲尔德大学）；Institute of Computing and Cybersystems, Michigan Technological University（密歇根理工大学计算与网络系统研究所）；Department of Computer Science, Michigan Technological University（密歇根理工大学计算机科学系）
HUPE: Heuristic Underwater Perceptual Enhancement with Semantic Collaborative Learning
School of Software Engineering, Dalian University of Technology（大连理工大学软件学院）；College of Information Science and Technology, Dalian Maritime University（大连海事大学信息科学技术学院）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
Five A$^{+}$ Network: You Only Need 9K Parameters for Underwater Image Enhancement
National University of Singapore（新加坡国立大学）；ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC研究院）；Fujian Provincial Key Laboratory of Oceanic Information Perception and Intelligent Processing（福建省海洋信息感知与智能处理重点实验室）
Open Robotics（开放机器人组织）；Naval Postgraduate School（海军研究生院）
We present DAVE Aquatic Virtual Environment (DAVE), an open source simulation stack for underwater robots, sensors, and environments. Conventional robotics simulators are not designed to address unique challenges that come with the marine environment, including but not limited to environment conditions that vary spatially and temporally, impaired or challenging perception, and the unavailability of data in a generally unexplored environment. Given the variety of sensors and platforms, wheels are often reinvented for specific use cases that inevitably resist wider adoption. Building on existing…
Style-Decoupled Adaptive Routing Network for Underwater Image Enhancement
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（测绘遥感信息工程国家重点实验室，武汉大学）；Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University（航空工程系，香港理工大学）
Queensland University of Technology（昆士兰科技大学）；Data61, CSIRO（联邦科学与工业研究组织数据61研究所）
Monitoring coral reefs using underwater vehicles increases the range of marine surveys and availability of historical ecological data by collecting significant quantities of images. Analysis of this imagery can be automated using a model trained to perform semantic segmentation, however it is too costly and time-consuming to densely label images for training supervised models. In this letter, we leverage photo-quadrat imagery labeled by ecologists with sparse point labels. We propose a point label aware method for propagating labels within superpixel regions to obtain augmented ground truth fo…
The underwater environment poses a complex problem for developing autonomous capabilities for Underwater Vehicle Manipulator Systems (UVMSs). The modeling of UVMSs is a complicated and costly process due to the highly nonlinear dynamics and the presence of unknown hydrodynamical effects. This is aggravated in tasks where the manipulation of objects is necessary, as this may not only introduce external disturbances that can lead to a fast degradation of the control system performance, but also requires the coordinating with a vision system for the correct grasping and operation of the object. I…
Downstream Task Inspired Underwater Image Enhancement: A Perception-Aware Study from Dataset Construction to Network Design
State Key Laboratory of Physical Oceanography, Ocean University of China（中国海洋大学物理海洋国家重点实验室）；Department of Electrical and Computer Engineering, Mississippi State University（密苏里州立大学电子与计算机工程系）
Detecting Endangered Marine Species in Autonomous Underwater Vehicle Imagery Using Point Annotations and Few-Shot Learning
Australian Centre for Robotics, University of Sydney（悉尼大学澳大利亚机器人中心）；Marine Technology Department, Norwegian University of Science and Technology（挪威科技大学海洋技术系）；Institute for Marine and Antarctic Studies, University of Tasmania（塔斯马尼亚大学海洋与南极研究所）
