Due to the unique characteristics of underwater environments, accurate 3D reconstruction of underwater objects poses a challenging problem in tasks such as underwater exploration and mapping. Traditional methods that rely on multiple sensor data for 3D reconstruction are time-consuming and face challenges in data acquisition in underwater scenarios. We propose UW-SDF, a framework for reconstructing target objects from multi-view underwater images based on neural SDF. We introduce hybrid geometric priors to optimize the reconstruction process, markedly enhancing the quality and efficiency of neural SDF reconstruction. Additionally, to address the challenge of segmentation consistency in multi-view images, we propose a novel few-shot multi-view target segmentation strategy using the general-purpose segmentation model (SAM), enabling rapid automatic segmentation of unseen objects. Through extensive qualitative and quantitative experiments on diverse datasets, we demonstrate that our proposed method outperforms the traditional underwater 3D reconstruction method and other neural rendering approaches in the field of underwater 3D reconstruction.
Depth estimation is critical for any robotic system. In the past years estimation of depth from monocular images have shown great improvement, however, in the underwater environment results are still lagging behind due to appearance changes caused by the medium. So far little effort has been invested on overcoming this. Moreover, underwater, there are more limitations for using high resolution depth sensors, this makes generating ground truth for learning methods another enormous obstacle. So far unsupervised methods that tried to solve this have achieved very limited success as they relied on…
In the context of robotic underwater operations, the visual degradations induced by the medium properties make difficult the exclusive use of cameras for localization purpose. Hence, most localization methods are based on expensive navigational sensors associated with acoustic positioning. On the other hand, visual odometry and visual SLAM have been exhaustively studied for aerial or terrestrial applications, but state-of-the-art algorithms fail underwater. In this paper we tackle the problem of using a simple low-cost camera for underwater localization and propose a new monocular visual odome…
This paper reports on WaterGAN, a generative adversarial network (GAN) for generating realistic underwater images from in-air image and depth pairings in an unsupervised pipeline used for color correction of monocular underwater images. Cameras onboard autonomous and remotely operated vehicles can capture high resolution images to map the seafloor, however, underwater image formation is subject to the complex process of light propagation through the water column. The raw images retrieved are characteristically different than images taken in air due to effects such as absorption and scattering,…
Refining raw disparity maps from different algorithms to exploit their complementary advantages is still challenging. Uncertainty estimation and complex disparity relationships among pixels limit the accuracy and robustness of existing methods and there is no standard method for fusion of different kinds of depth data. In this paper, we introduce a new method to fuse disparity maps from different sources, while incorporating supplementary information (intensity, gradient, etc.) into a refiner network to better refine raw disparity inputs. A discriminator network classifies disparities at diffe…
Investigation of the Challenges of Underwater-Visual-Monocular-SLAM
Oceanic Machine Vision, GEOMAR Helmholtz Centre for Ocean Research Kiel（海洋机器视觉，基尔海洋研究所）；Marine Data Science, Department of Computer Science, Christian-Albrechts-Universität zu Kiel（海洋数据科学，计算机科学系，基尔亚历山大-冯-洪堡大学）；Computer Vision and Robotics Research Institute (VICOROB), University of Girona（计算机视觉与机器人研究 institute（VICOROB），加泰罗尼亚大学）
Monocular scene reconstruction from posed images is challenging due to the complexity of a large environment. Recent volumetric methods learn to directly predict the TSDF volume and have demonstrated promising results in this task. However, most methods focus on how to extract and fuse the 2D features to a 3D feature volume, but none of them improve the way how the 3D volume is aggregated. In this work, we propose an SDF transformer network, which replaces the role of 3D CNN for better 3D feature aggregation. To reduce the explosive computation complexity of the 3D multi-head attention, we pro…
AquaFlow: A Monocular Gaussian Splatting SLAM for Underwater Streaming Reconstruction
Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）
Tree-Mamba: A Tree-Aware Mamba for Underwater Monocular Depth Estimation
Key Laboratory of Knowledge Automation for Industrial Processes, Ministry of Education（知识自动化工业过程关键实验室，教育部）；School of Automation and Electrical Engineering, University of Science and Technology Beijing（北京科技大学自动化与电气工程学院）；School of Ophthalmology and Optometry, Wenzhou Medical University（温州医科大学眼视光学院）；Department of Automation, Tsinghua University（清华大学自动化系）；Deepinfar Ocean Technology Inc.（深海远航技术有限公司）；School of Data Science, Lingnan University（岭南大学数据科学学院）；College of Computer Science, Nankai University（南开大学计算机学院）
In this paper, we present a fast monocular depth estimation method for enabling 3D perception capabilities of low-cost underwater robots. We formulate a novel end-to-end deep visual learning pipeline named UDepth, which incorporates domain knowledge of image formation characteristics of natural underwater scenes. First, we adapt a new input space from raw RGB image space by exploiting underwater light attenuation prior, and then devise a least-squared formulation for coarse pixel-wise depth prediction. Subsequently, we extend this into a domain projection loss that guides the end-to-end learni…
University of Michigan（密歇根大学）；DROP lab（DROP实验室）；Department of Naval Architecture and Marine Engineering（船舶与海洋工程系）
There has been much recent interest in deep learning methods for monocular image based object pose estimation. While object pose estimation is an important problem for autonomous robot interaction with the physical world, and the application space for monocular-based methods is expansive, there has been little work on applying these methods with fisheye imaging systems. Also, little exists in the way of annotated fisheye image datasets on which these methods can be developed and tested. The research landscape is even more sparse for object detection methods applied in the underwater domain, fi…
Visibility underwater is challenging, and degrades as the distance between the subject and camera increases, making vision tasks in the forward-looking direction more difficult. We have collected underwater forward-looking stereo-vision and visual-inertial image sets in the Mediterranean and Red Sea. To our knowledge there are no other public datasets in the underwater environment acquired with this camera-sensor orientation published with ground-truth. These datasets are critical for the development of several underwater applications, including obstacle avoidance, visual odometry, 3D tracking…
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所（蒂宾根））；University of Tübingen（蒂宾根大学）
In this paper, we aim to create generalizable and controllable neural signed distance fields (SDFs) that represent clothed humans from monocular depth observations. Recent advances in deep learning, especially neural implicit representations, have enabled human shape reconstruction and controllable avatar generation from different sensor inputs. However, to generate realistic cloth deformations from novel input poses, watertight meshes or dense full-body scans are usually needed as inputs. Furthermore, due to the difficulty of effectively modeling pose-dependent cloth deformations for diverse …
Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research Univ.（巴黎文理研究大学）；Deepmind（深度思维）
Recent work achieved impressive progress towards joint reconstruction of hands and manipulated objects from monocular color images. Existing methods focus on two alternative representations in terms of either parametric meshes or signed distance fields (SDFs). On one side, parametric models can benefit from prior knowledge at the cost of limited shape deformations and mesh resolutions. Mesh models, hence, may fail to precisely reconstruct details such as contact surfaces of hands and objects. SDF-based methods, on the other side, can represent arbitrary details but are lacking explicit priors.…
CASA-SDF: Curriculum-Aware Spatial Adaptation with Curvature-Guided Density for Neural Implicit Surface Reconstruction
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Automation, Nanjing University of Science and Technology（南京理工大学自动化学院）
Ocean4D: Generative Underwater 4D Reconstruction via Medium-Aware Video Diffusion
Faculty of Information Science and Engineering, Ocean University of China（中国海洋大学信息科学与工程学部）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
HI-SLAM: Monocular Real-time Dense Mapping with Hybrid Implicit Fields
University of Stuttgart（斯图加特大学）；Huawei Munich Research Center（华为慕尼黑研究中心）；Central Media Technology Institute, Huawei 2012 Laboratories（华为2012实验室中央媒体技术研究院）；Technical University of Munich（慕尼黑工业大学）
Learning Neural Parametric 3D Breast Shape Models for Metrical Surface Reconstruction From Monocular RGB Videos
Regensburg Medical Image Computing (ReMIC), OTH Regensburg（雷根斯堡应用技术大学雷根斯堡医学图像计算中心）；Department for Plastic, Hand and Reconstructive Surgery, University Hospital Regensburg（雷根斯堡大学医院整形、手与重建外科）
We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. To decouple the learning of underlying scene geometry from dynamic motion, we represent the scene as a time-invariant signed distance function (SDF) which serves as a reference frame, along with a time-conditioned deformation field. We further bridge this neural geometry representation with a differentiable physics simulator by designing a two-way conversion between the neural field and its corresponding hexahedral mesh, enabling us to estimate physics parameters from t…
Occlusion-Aware 3D Hand-Object Pose Estimation with Masked AutoEncoders
National Engineering Research Center for Robot Visual Perception and Control Technology, School of Artificial Intelligence and Robotics, Hunan University（国家机器人视觉感知与控制技术工程研究中心，人工智能与机器人学院，湖南大学）；School of Architecture and Art, Central South University（建筑与艺术学院，中南大学）；Department of Computer Science and Software Engineering, The University of Western Australia（计算机科学与软件工程系，西澳大学）
BALTIC: A Benchmark and Cross-Domain Strategy for 3D Reconstruction Across Air and Underwater Domains Under Varying Illumination
School of Engineering & Physical Sciences, Heriot-Watt University（赫瑞瓦德大学工程与物理科学学院）；Marine Data Science Group, Christian Albrechts University of Kiel（基尔大学海洋数据科学组）；Frontier Robotics, The National Robotarium（前沿机器人，国家机器人中心）
MaskHOI: Robust 3D Hand-Object Interaction Estimation via Masked Pre-training
PCA Lab, School of Intelligence Science and Technology, Nanjing University（PCA实验室，智能科学与技术学院，南京大学）；Nanyang Technological University（南洋理工大学）；Nanjing University of Science and Technology（南京理工大学）；Nankai University（南开大学）
ETH Zürich（苏黎世联邦理工学院）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；Lund University（隆德大学）；Zhejiang University（浙江大学）；University of Amsterdam（阿姆斯特丹大学）；University of Tübingen（图宾根大学）；Tübingen AI Center（图宾根人工智能中心）；Microsoft（微软）
Neural implicit representations have recently become popular in simultaneous localization and mapping (SLAM), especially in dense visual SLAM. However, previous works in this direction either rely on RGB-D sensors, or require a separate monocular SLAM approach for camera tracking and do not produce high-fidelity dense 3D scene reconstruction. In this paper, we present NICER-SLAM, a dense RGB SLAM system that simultaneously optimizes for camera poses and a hierarchical neural implicit map representation, which also allows for high-quality novel view synthesis. To facilitate the optimization pro…
LinStereo: Linear-Complexity Global Attention for Multi-Scale Iterative Stereo Matching
Australian Centre for Robotics (ACFR), School of Aerospace, Mechanical and Mechatronic Engineering (AMME), Faculty of Engineering, The University of Sydney（悉尼大学工程学院航空航天、机械与机电工程学院澳大利亚机器人中心（ACFR））
Due to the visual ambiguity, purely kinematic formulations on monocular human motion capture are often physically incorrect, biomechanically implausible, and can not reconstruct accurate interactions. In this work, we focus on exploiting the high-precision and non-differentiable physics simulator to incorporate dynamical constraints in motion capture. Our key-idea is to use real physical supervisions to train a target pose distribution prior for sampling-based motion control to capture physically plausible human motion. To obtain accurate reference motion with terrain interactions for the samp…
