This paper addresses real-time dense 3D reconstruction for a resource-constrained Autonomous Underwater Vehicle (AUV). Underwater vision-guided operations are among the most challenging as they combine 3D motion in the presence of external forces, limited visibility, and absence of global positioning. Obstacle avoidance and effective path planning require online dense reconstructions of the environment. Autonomous operation is central to environmental monitoring, marine archaeology, resource utilization, and underwater cave exploration. To address this problem, we propose to use SVIn2, a robust VIO method, together with a real-time 3D reconstruction pipeline. We provide extensive evaluation on four challenging underwater datasets. Our pipeline produces comparable reconstruction with that of COLMAP, the state-of-the-art offline 3D reconstruction method, at high frame rates on a single CPU.
French Research Institute for Exploitation of the Sea (IFREMER)（法国海洋开发研究院）；Research Centre in Underwater Robotics (CIRS), University of Girona（赫罗纳大学水下机器人研究中心）
Hyperspectral imaging has been increasingly used for underwater survey applications over the past years. As many hyperspectral cameras work as push-broom scanners, their use is usually limited to the creation of photo-mosaics based on a flat surface approximation and by interpolating the camera pose from dead-reckoning navigation. Yet, because of drift in the navigation and the mostly wrong flat surface assumption, the quality of the obtained photo-mosaics is often too low to support adequate analysis.In this paper we present an initial method for creating hyperspectral 3D reconstructions of u…
Queensland University of Technology（昆士兰科技大学）；The Australian Institute of Marine Science（澳大利亚海洋科学研究所）
This paper presents a novel approach to underwater terrain mapping for Autonomous Underwater Vehicles (AUVs) operating in close proximity to complex 3D environments. The proposed methodology creates a probabilistic elevation map of the terrain using a monocular image learning-based scene range estimator as a sensor. This scene range estimator can filter transient objects such as fish and lighting variations. The mapping approach considers uncertainty in both the estimated scene range and robot pose as the AUV moves through the environment. The resulting elevation map can be used for reactive p…
LIS UMR 7020, Aix-Marseille Université, CNRS, ENSAM, Université De Toulon（艾克斯-马赛大学、法国国家科学研究中心、国立高等工程技术学院、土伦大学 LIS UMR 7020 实验室）；D Optical Metrology (3DOM) unit, Bruno Kessler Foundation (FBK)（布鲁诺·凯斯勒基金会三维光学计量单元）
Although fully autonomous mapping methods are becoming more and more common and reliable, still the human operator is regularly employed in many 3D surveying missions. In a number of underwater applications, divers or pilots of remotely operated vehicles (ROVs) are still considered irreplaceable, and tools for real-time visualization of the mapped scene are essential to support and maximize the navigation and surveying efforts. For underwater exploration, image mosaicing has proved to be a valid and effective approach to visualize large mapped areas, often employed in conjunction with autonomo…
Visibility underwater is challenging, and degrades as the distance between the subject and camera increases, making vision tasks in the forward-looking direction more difficult. We have collected underwater forward-looking stereo-vision and visual-inertial image sets in the Mediterranean and Red Sea. To our knowledge there are no other public datasets in the underwater environment acquired with this camera-sensor orientation published with ground-truth. These datasets are critical for the development of several underwater applications, including obstacle avoidance, visual odometry, 3D tracking…
In this work, we exploit a depth estimation Fully Convolutional Residual Neural Network (FCRN) for in-air perspective images to estimate the depth of underwater perspective and omni-directional images. We train one conventional and one spherical FCRN for underwater perspective and omni-directional images, respectively. The spherical FCRN is derived from the perspective FCRN via a spherical longitude-latitude mapping. For that, the omni-directional camera is modeled as a sphere, while images captured by it are displayed in the longitude-latitude form. Due to the lack of underwater datasets, we …
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Depth estimation from a single underwater image is one of the most challenging problems and is highly ill-posed. Due to the absence of large generalized underwater depth datasets and the difficulty in obtaining ground truth depth-maps, supervised learning techniques such as direct depth regression cannot be used. In this paper, we propose an unsupervised method for depth estimation from a single underwater image taken `in the wild' by using haze as a cue for depth. Our approach is based on indirect depth-map estimation where we learn the mapping functions between unpaired RGB-D terrestrial ima…
Digital Twins Below the Surface: Enhancing Underwater Teleoperation
School of Engineering and Physical Sciences, Heriot-Watt University（赫瑞瓦特大学工程与物理科学学院）；School of Mathematics and Computer Sciences, Heriot-Watt University（赫瑞瓦特大学数学与计算机科学学院）
AquaFlow: A Monocular Gaussian Splatting SLAM for Underwater Streaming Reconstruction
Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）
Zhejiang University（浙江大学）；Shenzhen Sensetime Technology CO.,LTD（深圳商汤科技有限公司）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Autonomous Underwater Vehicle-Manipulator systems (AUVMS) is a new tool for ocean exploration, the AUVMS path planning problem is addressed in this paper. AUVMS is a high dimension system with a large difference in inertia distribution, also it works in a complex environment with obstacles. By integrating the rapidly-exploring random tree(RRT) algorithm with the AUVMS kinematics model, the proposed RRTAUVMS algorithm could randomly sample in the configuration space(C-Space), and also grow the tree directly towards the workspace goal in the task space. The RRTAUVMS can also deal with the redund…
Mapping the seafloor with underwater imaging cameras is of significant importance for various applications including marine engineering, geology, geomorphology, archaeology and biology. For shallow waters, among the underwater imaging challenges, caustics i.e., the complex physical phenomena resulting from the projection of light rays being refracted by the wavy surface, is likely the most crucial one. Caustics is the main factor during underwater imaging campaigns that massively degrade image quality and affect severely any 2D mosaicking or 3D reconstruction of the seabed. In this work, we pr…
Dyson Technology Ltd.（戴森技术有限公司）；Imperial College London（帝国理工学院）
We propose a novel dense mapping framework for sparse visual SLAM systems which leverages a compact scene representation. State-of-the-art sparse visual SLAM systems provide accurate and reliable estimates of the camera trajectory and locations of landmarks. While these sparse maps are useful for localization, they cannot be used for other tasks such as obstacle avoidance or scene understanding. In this paper we propose a dense mapping framework to complement sparse visual SLAM systems which takes as input the camera poses, keyframes and sparse points produced by the SLAM system and predicts a…
In this paper we present a complete framework for Underwater SLAM utilizing a single inexpensive sensor. Over the recent years, imaging technology of action cameras is producing stunning results even under the challenging conditions of the underwater domain. The GoPro 9 camera provides high definition video in synchronization with an Inertial Measurement Unit (IMU) data stream encoded in a single mp4 file. The visual inertial SLAM framework is augmented to adjust the map after each loop closure. Data collected at an artificial wreck of the coast of South Carolina and in caverns and caves in Fl…
Technical University of Munich（慕尼黑工业大学）；Artisense；Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用科学大学）
In this paper, we present TANDEM a real-time monocular tracking and dense mapping framework. For pose estimation, TANDEM performs photometric bundle adjustment based on a sliding window of keyframes. To increase the robustness, we propose a novel tracking front-end that performs dense direct image alignment using depth maps rendered from a global model that is built incrementally from dense depth predictions. To predict the dense depth maps, we propose Cascade View-Aggregation MVSNet (CVA-MVSNet) that utilizes the entire active keyframe window by hierarchically constructing 3D cost volumes wit…
Cutting-edge 3D reconstruction solutions for underwater coral reef images: A review and comparison
State Key Laboratory of Information Engineering in Surveying Mapping and Remote Sensing（信息工程测绘与遥感国家重点实验室）；Wuhan University（武汉大学）；Institute of Geodesy and Photogrammetry（大地测量与摄影测量研究所）；ETH Zurich（苏黎世联邦理工学院）；The Hong Kong Polytechnic University（香港理工大学）；Peking University（北京大学）
HGS-Mapping: Online Dense Mapping Using Hybrid Gaussian Representation in Urban Scenes
Fudan University（复旦大学）；School of Future Technology, Harbin Institute of Technology（哈尔滨工业大学未来技术学院）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）
OG-Mapping: Octree-based Structured 3D Gaussians for Online Dense Mapping
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；PengCheng Laboratory（鹏城实验室）；School of Computer Science and Technology, Shandong University（山东大学计算机科学与技术学院）；Beijing Technology and Business University（北京工商大学）
University of Birmingham（伯明翰大学）；Toshiba Research Europe（东芝欧洲研究院）
For intelligent robotics applications, extending 3D mapping to 3D semantic mapping enables robots to, not only localize themselves with respect to the scene's geometrical features but also simultaneously understand the higher level meaning of the scene contexts. Most previous methods focus on geometric 3D reconstruction and scene understanding independently notwithstanding the fact that joint estimation can boost the accuracy of the semantic mapping. In this paper, a dense RGB-D semantic mapping system with a Pixel-Voxel network is proposed, which can perform dense 3D mapping while simultaneou…
Australian Centre for Field Robotics（澳大利亚野外机器人中心）；The University of Sydney（悉尼大学）
This paper describes the design, implementation and testing of a suite of algorithms to enable depth constrained autonomous bathymetric (underwater topography) mapping by an Autonomous Surface Vessel (ASV). Given a target depth and a bounding polygon, the ASV will find and follow the intersection of the bounding polygon and the depth contour as modeled online with a Gaussian Process (GP). This intersection, once mapped, will then be used as a boundary within which a path will be planned for coverage to build a map of the Bathymetry. Methods for sequential updates to GP's are described allowing…
Kyushu University（九州大学）；Hiroshima City University（广岛市立大学）
Dense 3D shape acquisition of swimming human or live fish is an important research topic for sports, biological science and so on. For this purpose, active stereo sensor is usually used in the air, however it cannot be applied to the underwater environment because of refraction, strong light attenuation and severe interference of bubbles. Passive stereo is a simple solution for capturing dynamic scenes at underwater environment, however the shape with textureless surfaces or irregular reflections cannot be recovered. Recently, the stereo camera pair with a pattern projector for adding artifici…
University of South Carolina（南卡罗来纳大学）；McGill University（麦吉尔大学）；Dartmouth College（达特茅斯学院）
In this paper, we propose a real-time deep learning approach for determining the 6D relative pose of Autonomous Underwater Vehicles (AUV) from a single image. A team of autonomous robots localizing themselves in a communication-constrained underwater environment is essential for many applications such as underwater exploration, mapping, multi-robot convoying, and other multi-robot tasks. Due to the profound difficulty of collecting ground truth images with accurate 6D poses underwater, this work utilizes rendered images from the Unreal Game Engine simulation for training. An image-to-image tra…
Design, Implementation and Evaluation of an External Pose-Tracking System for Underwater Cameras
GEOMAR Helmholtz Centre for Ocean Research Kiel（基尔亥姆霍兹海洋研究中心）；Christian-Albrechts-Universität zu Kiel（基尔大学）；University of Applied Sciences Kiel（基尔应用科学大学）
Edith Cowan University（埃迪斯科文大学）；The University of Western Australia（西澳大学）
Monitoring underwater habitats is a vital part of observing the condition of the environment. The detection and mapping of underwater vegetation, especially seagrass has drawn the attention of the research community as early as the nineteen eighties. Initially, this monitoring relied on in situ observation by experts. Later, advances in remote-sensing technology, satellite-monitoring techniques and, digital photo- and video-based techniques opened a window to quicker, cheaper, and, potentially, more accurate seagrass-monitoring methods. So far, for seagrass detection and mapping, digital image…
In this paper, we propose a novel technique to reconstruct 3D surface of an underwater object using stereo images. Reconstructing the 3D surface of an underwater object is really a challenging task due to degraded quality of underwater images. There are various reason of quality degradation of underwater images i.e., non-uniform illumination of light on the surface of objects, scattering and absorption effects. Floating particles present in underwater produces Gaussian noise on the captured underwater images which degrades the quality of images. The degraded underwater images are preprocessed …
Investigation of the Challenges of Underwater-Visual-Monocular-SLAM
Oceanic Machine Vision, GEOMAR Helmholtz Centre for Ocean Research Kiel（海洋机器视觉，基尔海洋研究所）；Marine Data Science, Department of Computer Science, Christian-Albrechts-Universität zu Kiel（海洋数据科学，计算机科学系，基尔亚历山大-冯-洪堡大学）；Computer Vision and Robotics Research Institute (VICOROB), University of Girona（计算机视觉与机器人研究 institute（VICOROB），加泰罗尼亚大学）
Knowledge Distillation for Underwater Feature Extraction and Matching via GAN-synthesized Images
Department of Electrical and Electronic Engineering, The University of Melbourne（电气与电子工程系，墨尔本大学）；School of Mathematics and Statistics, The University of Melbourne（数学与统计学系，墨尔本大学）
University of Birmingham（伯明翰大学）；University of Lincoln（林肯大学）；Toshiba Research Europe（东芝欧洲研究院）
This paper introduces a fully deep learning approach to monocular SLAM, which can perform simultaneous localization using a neural network for learning visual odometry (L-VO) and dense 3D mapping. Dense 2D flow and a depth image are generated from monocular images by sub-networks, which are then used by a 3D flow associated layer in the L-VO network to generate dense 3D flow. Given this 3D flow, the dual-stream L-VO network can then predict the 6DOF relative pose and furthermore reconstruct the vehicle trajectory. In order to learn the correlation between motion directions, the Bivariate Gauss…
The paper presents a direct visual-inertial odometry system. In particular, a tightly coupled nonlinear optimization based method is proposed by integrating the recent advances in direct dense tracking and Inertial Measurement Unit (IMU) pre-integration, and a factor graph optimization is adapted to estimate the pose of the camera and rebuild a semi-dense map. Two sliding windows are maintained in the proposed approach. The first one, based on Direct Sparse Odometry (DSO), is to estimate the depths of candidate points for mapping and dense visual tracking. In the second one, measurements from …
Generating 3D point cloud (PC) data from noisy sonar measurements is a problem that has potential applications for bathymetry mapping, artificial object inspection, mapping of aquatic plants and fauna as well as underwater navigation and localization of vehicles such as submarines. Side-scan sonar sensors are available in inexpensive cost ranges, especially in fish-finders, where the transducers are usually mounted to the bottom of a boat and can approach shallower depths than the ones attached to an Uncrewed Underwater Vehicle (UUV) can. However, extracting 3D information from side-scan sonar…
Corals are the primary habitat-building life-form on reefs that support a quarter of the species in the ocean. A coral reef ecosystem usually consists of reefs, each of which is like a tall building in any city. These reef-building corals secrete hard calcareous exoskeletons that give them structural rigidity, and are also a prerequisite for our accurate 3D modeling and semantic mapping using advanced photogrammetric computer vision and machine learning. Underwater videography as a modern underwater remote sensing tool is a high-resolution coral habitat survey and mapping technique. In this pa…
OPPO US Research Center, InnoPeak Technology（OPPO美国研究中心，InnoPeak科技）
We present a robust and accurate depth refinement system, named GeoRefine, for geometrically-consistent dense mapping from monocular sequences. GeoRefine consists of three modules: a hybrid SLAM module using learning-based priors, an online depth refinement module leveraging self-supervision, and a global mapping module via TSDF fusion. The proposed system is online by design and achieves great robustness and accuracy via: (i) a robustified hybrid SLAM that incorporates learning-based optical flow and/or depth; (ii) self-supervised losses that leverage SLAM outputs and enforce long-term geomet…
