Live fish recognition is one of the most crucial elements of fisheries survey applications where vast amount of data are rapidly acquired. Different from general scenarios, challenges to underwater image recognition are posted by poor image quality, uncontrolled objects and environment, as well as difficulty in acquiring representative samples. Also, most existing feature extraction techniques are hindered from automation due to involving human supervision. Toward this end, we propose an underwater fish recognition framework that consists of a fully unsupervised feature learning technique and an error-resilient classifier. Object parts are initialized based on saliency and relaxation labeling to match object parts correctly. A non-rigid part model is then learned based on fitness, separation and discrimination criteria. For the classifier, an unsupervised clustering approach generates a binary class hierarchy, where each node is a classifier. To exploit information from ambiguous images, the notion of partial classification is introduced to assign coarse labels by optimizing the "benefit" of indecision made by the classifier. Experiments show that the proposed framework achieves high accuracy on both public and self-collected underwater fish images with high uncertainty and class imbalance.
The target of this paper is to recommend a way for Automated classification of Fish species. A high accuracy fish classification is required for greater understanding of fish behavior in Ichthyology and by marine biologists. Maintaining a ledger of the number of fishes per species and marking the endangered species in large and small water bodies is required by concerned institutions. Majority of available methods focus on classification of fishes outside of water because underwater classification poses challenges such as background noises, distortion of images, the presence of other water bod…
Clean energy from oceans and rivers is becoming a reality with the development of new technologies like tidal and instream turbines that generate electricity from naturally flowing water. These new technologies are being monitored for effects on fish and other wildlife using underwater video. Methods for automated analysis of underwater video are needed to lower the costs of analysis and improve accuracy. A deep learning model, YOLO, was trained to recognize fish in underwater video using three very different datasets recorded at real-world water power sites. Training and testing with examples…
University of Washington（华盛顿大学）；National Taipei University of Technology（国立台北科技大学）；Alaska Fisheries Science Center（阿拉斯加渔业科学中心）
Fishery surveys that call for the use of single or multiple underwater cameras have been an emerging technology as a non-extractive mean to estimate the abundance of fish stocks. Tracking live fish in an open aquatic environment posts challenges that are different from general pedestrian or vehicle tracking in surveillance applications. In many rough habitats fish are monitored by cameras installed on moving platforms, where tracking is even more challenging due to inapplicability of background models. In this paper, a novel tracking algorithm based on the deformable multiple kernels (DMK) is …
Marine scientists use remote underwater video recording to survey fish species in their natural habitats. This helps them understand and predict how fish respond to climate change, habitat degradation, and fishing pressure. This information is essential for developing sustainable fisheries for human consumption, and for preserving the environment. However, the enormous volume of collected videos makes extracting useful information a daunting and time-consuming task for a human. A promising method to address this problem is the cutting-edge Deep Learning (DL) technology.DL can help marine scien…
Overcoming Annotation Bottlenecks in Underwater Fish Segmentation: A Robust Self-Supervised Learning Approach
College of Science and Engineering, James Cook University（詹姆斯库克大学科学与工程学院）；ARC Research Hub for Supercharging Tropical Aquaculture through Genetic Solutions, James Cook University（詹姆斯库克大学ARC热带水产养殖基因解决方案研究中心）
Closer to Ground Truth: Realistic Shape and Appearance Labeled Data Generation for Unsupervised Underwater Image Segmentation
NORCE Norwegian Research Center AS（NORCE挪威研究中心）；National University of Science and Technology "Politehnica" Bucharest（布加勒斯特理工科学技术大学）；"Simion Stoilow" Institute of Mathematics of the Romanian Academy（罗马尼亚科学院西米翁斯托伊洛夫数学研究所）
Marine ecosystems and their fish habitats are becoming increasingly important due to their integral role in providing a valuable food source and conservation outcomes. Due to their remote and difficult to access nature, marine environments and fish habitats are often monitored using underwater cameras. These cameras generate a massive volume of digital data, which cannot be efficiently analysed by current manual processing methods, which involve a human observer. DL is a cutting-edge AI technology that has demonstrated unprecedented performance in analysing visual data. Despite its application…
James Cook University（詹姆斯·库克大学）；National Research Council of Italy（意大利国家研究委员会）
Given a sufficiently large training dataset, it is relatively easy to train a modern convolution neural network (CNN) as a required image classifier. However, for the task of fish classification and/or fish detection, if a CNN was trained to detect or classify particular fish species in particular background habitats, the same CNN exhibits much lower accuracy when applied to new/unseen fish species and/or fish habitats. Therefore, in practice, the CNN needs to be continuously fine-tuned to improve its classification accuracy to handle new project-specific fish species or habitats. In this work…
Griffith University（格里菲斯大学）；Australian Rivers Institute（澳大利亚河流研究所）；University of the Sunshine Coast（阳光海岸大学）
Uses of underwater videos to assess diversity and abundance of fish are being rapidly adopted by marine biologists. Manual processing of videos for quantification by human analysts is time and labour intensive. Automatic processing of videos can be employed to achieve the objectives in a cost and time-efficient way. The aim is to build an accurate and reliable fish detection and recognition system, which is important for an autonomous robotic platform. However, there are many challenges involved in this task (e.g. complex background, deformation, low resolution and light propagation). Recent a…
University of Washington（华盛顿大学）；Alaska Fisheries Science Center, National Oceanic and Atmospheric Administration（美国国家海洋和大气管理局阿拉斯加渔业科学中心）
Non-extractive fish abundance estimation with the aid of visual analysis has drawn increasing attention. Unstable illumination, ubiquitous noise and low frame rate video capturing in the underwater environment, however, make conventional tracking methods unreliable. In this paper, we present a multiple fish tracking system for low-contrast and low-frame-rate stereo videos with the use of a trawl-based underwater camera system. An automatic fish segmentation algorithm overcomes the low-contrast issues by adopting a histogram backprojection approach on double local-thresholded images to ensure a…
EPANet: Efficient Path Aggregation Network for Underwater Fish Detection
Dalian Key Laboratory of Smart Fishery（大连智能渔业重点实验室）；College of Information Engineering, Dalian Ocean University（大连海洋大学信息工程学院）；Key Laboratory of Environment Controlled Aquaculture（环境可控水产养殖重点实验室）；Dalian Polytechnic University School of International Education（大连职业技术学院国际教育学院）
James Cook University（詹姆斯库克大学）；University of British Columbia（不列颠哥伦比亚大学）；Element AI
Visual analysis of complex fish habitats is an important step towards sustainable fisheries for human consumption and environmental protection. Deep Learning methods have shown great promise for scene analysis when trained on large-scale datasets. However, current datasets for fish analysis tend to focus on the classification task within constrained, plain environments which do not capture the complexity of underwater fish habitats. To address this limitation, we present DeepFish as a benchmark suite with a large-scale dataset to train and test methods for several computer vision tasks. The da…
University of Michigan（密歇根大学）；DROP lab（DROP实验室）；Department of Naval Architecture and Marine Engineering（船舶与海洋工程系）
There has been much recent interest in deep learning methods for monocular image based object pose estimation. While object pose estimation is an important problem for autonomous robot interaction with the physical world, and the application space for monocular-based methods is expansive, there has been little work on applying these methods with fisheye imaging systems. Also, little exists in the way of annotated fisheye image datasets on which these methods can be developed and tested. The research landscape is even more sparse for object detection methods applied in the underwater domain, fi…
Aalborg University（奥尔堡大学）；Pioneer Center for AI（人工智能先锋中心）
There exist no publicly available annotated underwater multi-object tracking (MOT) datasets captured in turbid environments. To remedy this we propose the BrackishMOT dataset with focus on tracking schools of small fish, which is a notoriously difficult MOT task. BrackishMOT consists of 98 sequences captured in the wild. Alongside the novel dataset, we present baseline results by training a state-of-the-art tracker. Additionally, we propose a framework for creating synthetic sequences in order to expand the dataset. The framework consists of animated fish models and realistic underwater enviro…
University of Minnesota Twin Cities（明尼苏达大学双城分校）；Minnesota Robotics Institute（明尼苏达机器人研究所）
This paper presents a deep-learned facial recognition method for underwater robots to identify scuba divers. Specifically, the proposed method is able to recognize divers underwater with faces heavily obscured by scuba masks and breathing apparatus. Our contribution in this research is towards robust facial identification of individuals under significant occlusion of facial features and image degradation from underwater optical distortions. With the ability to correctly recognize divers, autonomous underwater vehicles (AUV) will be able to engage in collaborative tasks with the correct person …
Deep convolutional neural networks generally perform well in underwater object recognition tasks on both optical and sonar images. Many such methods require hundreds, if not thousands, of images per class to generalize well to unseen examples. However, obtaining and labeling sufficiently large volumes of data can be relatively costly and time-consuming, especially when observing rare objects or performing real-time operations. Few-Shot Learning (FSL) efforts have produced many promising methods to deal with low data availability. However, little attention has been given in the underwater domai…
Underwater automatic target recognition (UATR) has been a challenging research topic in ocean engineering. Although deep learning brings opportunities for target recognition on land and in the air, underwater target recognition techniques based on deep learning have lagged due to sensor performance and the size of trainable data. This letter proposed a framework for learning the visual representation of underwater acoustic imageries, which takes a transformer-based style transfer model as the main body. It could replace the low-level texture features of optical images with the visual features …
CIDCO - Development Center for Ocean Mapping（CIDCO——海洋测绘开发中心）
This paper details a new method to recognize and detect underwater objects in real-time sidescan sonar data imagery streams, with case-studies of applications for underwater archeology, and ghost fishing gear retrieval. We first synthesize images from sidescan data, apply geometric and radiometric corrections, then use 2D feature detection algorithms to identify point clouds of descriptive visual microfeatures such as corners and edges in the sonar images. We then apply a clustering algorithm on the feature point clouds to group feature sets into regions of interest, reject false positives, yi…
FinSight-Net:A Physics-Aware Decoupled Network with Frequency-Domain Compensation for Underwater Fish Detection in Smart Aquaculture
Dalian Key Laboratory of Smart Fishery（大连智能渔业重点实验室）；College of Information Engineering, Dalian Ocean University（大连海洋大学信息工程学院）；Key Laboratory of Environment Controlled Aquaculture（环境可控养殖重点实验室）；Dalian Polytechnic University School of International Education（大连职业技术学院国际教育学院）
This paper presents an approach for autonomous underwater robots to visually detect and identify divers. The proposed approach enables an autonomous underwater robot to detect multiple divers in a visual scene and distinguish between them. Such methods are useful for robots to identify a human leader, for example, in multi-human/robot teams where only designated individuals are allowed to command or lean a team of robots. Initial diver identification is performed using the Faster R-CNN algorithm with a region proposal network which produces bounding boxes around the divers' locations. Subseque…
Recurrent Neural Networks (RNNs) have had considerable success in classifying and predicting sequences. We demonstrate that RNNs can be effectively used in order to encode sequences and provide effective representations. The methodology we use is based on Fisher Vectors, where the RNNs are the generative probabilistic models and the partial derivatives are computed using backpropagation. State of the art results are obtained in two central but distant tasks, which both rely on sequences: video action recognition and image annotation. We also show a surprising transfer learning result from the …
Fuzhou University（福州大学）；Peng Cheng Laboratory（鹏城实验室）；Beijing University of Technology（北京工业大学）；Beijing Artificial Intelligence Institute（北京人工智能研究院）；Université de Nantes（南特大学）；Laboratoire des Sciences du Numérique de Nantes（南特数字科学实验室）
The widespread image applications have greatly promoted the vision-based tasks, in which the Image Quality Assessment (IQA) technique has become an increasingly significant issue. For user enjoyment in multimedia systems, the IQA exploits image fidelity and aesthetics to characterize user experience; while for other tasks such as popular object recognition, there exists a low correlation between utilities and perceptions. In such cases, the fidelity-based and aesthetics-based IQA methods cannot be directly applied. To address this issue, this paper proposes a utility-oriented IQA in object rec…
With the development of underwater object grabbing technology, underwater object recognition and segmentation of high accuracy has become a challenge. The existing underwater object detection technology can only give the general position of an object, unable to give more detailed information such as the outline of the object, which seriously affects the grabbing efficiency. To address this problem, we label and establish the first underwater semantic segmentation dataset of real scene(DUT-USEG:DUT Underwater Segmentation Dataset). The DUT- USEG dataset includes 6617 images, 1487 of which have …
AquaFeat+: an Underwater Vision Learning-based Enhancement Method for Object Detection, Classification, and Tracking
Centro de Ciências Computacionais (C3), Universidade Federal do Rio Grande (FURG), Brazil（巴西里约格兰德联邦大学计算科学研究中心（C3））；Instituto de Ciências Exatas e Tecnologia (ICET), Universidade Federal do Amazonas (UFAM), Brazil（巴西亚马逊联邦大学精确科学与技术研究所（ICET））
Thales（泰雷兹集团）；Thales Digital Solutions（泰雷兹数字解决方案公司）；INRIA（法国国家信息与自动化研究所）
Conventional convolution neural networks (CNNs) trained on narrow Field-of-View (FoV) images are the state-of-the-art approaches for object recognition tasks. Some methods proposed the adaptation of CNNs to ultra-wide FoV images by learning deformable kernels. However, they are limited by the Euclidean geometry and their accuracy degrades under strong distortions caused by fisheye projections. In this work, we demonstrate that learning the shape of convolution kernels in non-Euclidean spaces is better than existing deformable kernel methods. In particular, we propose a new approach that learns…
Universitat de Barcelona（巴塞罗那大学）；Universidade Federal de Santa Catarina（圣卡塔琳娜联邦大学）；Max Planck Institute of Animal Behavior（马克斯·普朗克动物行为研究所）；Universidade Federal do Paraná（巴拉那联邦大学）；Witwatersrand University（威特沃特斯兰德大学）；Oregon State University（俄勒冈州立大学）；University of Zurich（苏黎世大学）；Universitat Autonoma de Barcelona（巴塞罗那自治大学）
Effective conservation actions require effective population monitoring. However, accurately counting animals in the wild to inform conservation decision-making is difficult. Monitoring populations through image sampling has made data collection cheaper, wide-reaching and less intrusive but created a need to process and analyse this data efficiently. Counting animals from such data is challenging, particularly when densely packed in noisy images. Attempting this manually is slow and expensive, while traditional computer vision methods are limited in their generalisability. Deep learning is the …
Huazhong Institute of Electro-Optics（华中光电技术研究所）；Wuhan National Laboratory for Optoelectronics（武汉光电国家实验室）；Harbin Engineering University（哈尔滨工程大学）
In real-world underwater environment, exploration of seabed resources, underwater archaeology, and underwater fishing rely on a variety of sensors, vision sensor is the most important one due to its high information content, non-intrusive, and passive nature. However, wavelength-dependent light attenuation and back-scattering result in color distortion and haze effect, which degrade the visibility of images. To address this problem, firstly, we proposed an unsupervised generative adversarial network (GAN) for generating realistic underwater images (color distortion and haze effect) from in-air…
Image pattern recognition is an important area in digital image processing. An efficient pattern recognition algorithm should be able to provide correct recognition at a reduced computational time. Off late amongst the machine learning pattern recognition algorithms, Artificial fish swarm algorithm is one of the swarm intelligence optimization algorithms that works based on population and stochastic search. In order to achieve acceptable result, there are many parameters needs to be adjusted in AFSA. Among these parameters, visual and step are very significant in view of the fact that artifici…
Calibration and Comparative Analysis of Forward-Looking Sonar and 3D Sonar for Enhanced Underwater Object Recognition
College of Engineering at the University of Florida（佛罗里达大学工程学院）；Seoul National University（首尔大学）；University of South Florida（南佛罗里达大学）；University of Miami（迈阿密大学）
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Federal University of Bahia (UFBA)（巴伊亚联邦大学（UFBA））
Under the sea, visible spectrum cameras have limited sensing capacity, being able to detect objects only in clear water, but in a constrained range. Considering any sea water condition, sonars are more suitable to support autonomous underwater vehicles' navigation, even in turbid condition. Despite that sonar suitability, this type of sensor does not provide high-density information, such as optical sensors, making the process of object recognition to be more complex. To deal with that problem, we propose a novel trainable method to detect and recognize (identify) specific target objects under…
With the growth of sensing, control and robotic technologies, autonomous underwater vehicles (AUVs) have become useful assistants to human divers for performing various underwater operations. In the current practice, the divers are required to carry expensive, bulky, and waterproof keyboards or joystick-based controllers for supervision and control of AUVs. Therefore, diver action-based supervision is becoming increasingly popular because it is convenient, easier to use, faster, and cost effective. However, the various environmental, diver and sensing uncertainties present underwater makes it …
On-board classification of underwater images using hybrid classical-quantum CNN based method
1 Department of Physics, Mahindra University, Hyderabad, 500043, Telangana, India；2 Department of Computer Science, Mahindra University, Hyderabad, 500043, Telangana, India；3 Department of Mechanical \& Aerospace Engineering, Mahindra University, Hyderabad, 500043, Telangana, India
Object detection is a very important basic research direction in the field of computer vision and a basic method for other advanced tasks in the field of computer vision. It has been widely used in practical applications such as object tracking, video behavior recognition and underwater robotics vision. The Cascade-RCNN and Deformable Convolution Network are both classical and excellent object detection algorithms. In this report, we evaluate our Cascade-DCN based method on underwater optical image and acoustics image datasets with different engineering tricks and augumentation.
End-to-end Learning of a Fisher Vector Encoding for Part Features in Fine-grained Recognition
Friedrich-Schiller-University Jena（耶拿弗里德里希·席勒大学）；Michael Stifel Center Jena for Data-Driven and Simulation Science（耶拿数据驱动与模拟科学迈克尔·施蒂费尔中心）；German Aerospace Center (DLR)（德国航空航天中心（DLR））
Downstream Task Inspired Underwater Image Enhancement: A Perception-Aware Study from Dataset Construction to Network Design
State Key Laboratory of Physical Oceanography, Ocean University of China（中国海洋大学物理海洋国家重点实验室）；Department of Electrical and Computer Engineering, Mississippi State University（密苏里州立大学电子与计算机工程系）
Peking University（北京大学）；Chongqing University of Technology（重庆理工大学）；Peng Cheng Laboratory（鹏城实验室）
The performance of existing underwater object detection methods degrades seriously when facing domain shift caused by complicated underwater environments. Due to the limitation of the number of domains in the dataset, deep detectors easily memorize a few seen domains, which leads to low generalization ability. There are two common ideas to improve the domain generalization performance. First, it can be inferred that the detector trained on as many domains as possible is domain-invariant. Second, for the images with the same semantic content in different domains, their hidden features should be…
Norwegian University of Science and Technology（挪威科技大学）；EXPOSED Aquaculture Research Centre（EXPOSED水产养殖研究中心）
Identifying individual salmon can be very beneficial for the aquaculture industry as it enables monitoring and analyzing fish behavior and welfare. For aquaculture researchers identifying individual salmon is imperative to their research. The current methods of individual salmon tagging and tracking rely on physical interaction with the fish. This process is inefficient and can cause physical harm and stress for the salmon. In this paper we propose FishNet, based on a deep learning technique that has been successfully used for identifying humans, to identify salmon.We create a dataset of label…
Singapore University of Technology and Design（新加坡科技设计大学）；Institute for Infocomm Research（新加坡资讯通信研究院）
With the increasing availability of wearable devices, research on egocentric activity recognition has received much attention recently. In this paper, we build a Multimodal Egocentric Activity dataset which includes egocentric videos and sensor data of 20 fine-grained and diverse activity categories. We present a novel strategy to extract temporal trajectory-like features from sensor data. We propose to apply the Fisher Kernel framework to fuse video and temporal enhanced sensor features. Experiment results show that with careful design of feature extraction and fusion algorithm, sensor data c…
We presents in this paper a novel fish classification methodology based on a combination between robust feature selection, image segmentation and geometrical parameter techniques using Artificial Neural Network and Decision Tree. Unlike existing works for fish classification, which propose descriptors and do not analyze their individual impacts in the whole classification task and do not make the combination between the feature selection, image segmentation and geometrical parameter, we propose a general set of features extraction using robust feature selection, image segmentation and geometri…
DTU Electrical Engineering Technical University of Denmark（丹麦技术大学电气工程学院）；AUV Competence Centre ATLAS MARIDAN ApS（ATLAS MARIDAN自主水下航行器能力中心有限公司）
Low-bandwidth communication, such as underwater acoustic communication, is limited by best-case data rates of 30--50 kbit/s. This renders such channels unusable or inefficient at best for single image, video, or other bandwidth-demanding sensor-data transmission. To combat data-transmission bottlenecks, we consider practical use-cases within the maritime domain and investigate the prospect of Single Image Super-Resolution methodologies. This is investigated on a large, diverse dataset obtained during years of trawl fishing where cameras have been placed in the fishing nets. We propose down-sam…
