The research on neural radiance fields for new view synthesis has experienced explosive growth with the development of new models and extensions. The NERF algorithm, suitable for underwater scenes or scattering media, is also evolving. Existing underwater 3D reconstruction systems still face challenges such as extensive training time and low rendering efficiency. This paper proposes an improved underwater 3D reconstruction system to address these issues and achieve rapid, high-quality 3D reconstruction.To begin with, we enhance underwater videos captured by a monocular camera to correct the poor image quality caused by the physical properties of the water medium while ensuring consistency in enhancement across adjacent frames. Subsequently, we perform keyframe selection on the video frames to optimize resource utilization and eliminate the impact of dynamic objects on the reconstruction results. The selected keyframes, after pose estimation using COLMAP, undergo a three-dimensional reconstruction improvement process using neural radiance fields based on multi-resolution hash coding for model construction and rendering.
Technical University of Munich（慕尼黑工业大学）；Max Planck Institute for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所（蒂宾根））；Apple（苹果公司）
3D reconstruction of large scenes is a challenging problem due to the high-complexity nature of the solution space, in particular for generative neural networks. In contrast to traditional generative learned models which encode the full generative process into a neural network and can struggle with maintaining local details at the scene level, we introduce a new method that directly leverages scene geometry from the training database. First, we learn to synthesize an initial estimate for a 3D scene, constructed by retrieving a top-k set of volumetric chunks from the scene database. These candi…
In this work, we propose a method for three-dimensional (3D) reconstruction of wide crime scene, based on a Simultaneous Localization and Mapping (SLAM) approach. We used a Kinect V2 Time-of-Flight (TOF) RGB-D camera to provide colored dense point clouds at a 30 Hz frequency. This device is moved freely (6 degrees of freedom) during the scene exploration. The implemented SLAM solution aligns successive point clouds using an 3D keypoints description and matching approach. This type of approach exploits both colorimetric and geometrical information, and permits reconstruction under poor illumina…
Stanford University（斯坦福大学）；Princeton University（普林斯顿大学）；Technical University of Munich（慕尼黑工业大学）
A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available -- current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surfa…
Camera motion estimation is a key technique for 3D scene reconstruction and Simultaneous localization and mapping (SLAM). To make it be feasibly achieved, previous works usually assume slow camera motions, which limits its usage in many real cases. We propose an end-to-end 3D reconstruction system which combines color, depth and inertial measurements to achieve robust reconstruction with fast sensor motions. Our framework extends Kalman filter to fuse the three kinds of information and involve an iterative method to jointly optimize feature correspondences, camera poses and scene geometry. We …
BNRist（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；Beijing Normal University（北京师范大学）；City University of Hong Kong（香港城市大学）；Tencent PCG（腾讯平台与内容事业群）
High-fidelity 3D scene reconstruction from monocular videos continues to be challenging, especially for complete and fine-grained geometry reconstruction. The previous 3D reconstruction approaches with neural implicit representations have shown a promising ability for complete scene reconstruction, while their results are often over-smooth and lack enough geometric details. This paper introduces a novel neural implicit scene representation with volume rendering for high-fidelity online 3D scene reconstruction from monocular videos. For fine-grained reconstruction, our key insight is to incorpo…
Holistic Large-Scale Scene Reconstruction via Mixed Gaussian Splatting
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Electronic Information, Wuhan University（武汉大学电子信息学院）；State Key Lab. of LIESMARS, Wuhan University（武汉大学LIESMARS国家重点实验室）；Institute for Math & AI, Wuhan University（武汉大学数学与人工智能研究院）
Reconstructing general dynamic scenes is important for many computer vision and graphics applications. Recent works represent the dynamic scene with neural radiance fields for photorealistic view synthesis, while their surface geometry is under-constrained and noisy. Other works introduce surface constraints to the implicit neural representation to disentangle the ambiguity of geometry and appearance field for static scene reconstruction. To bridge the gap between rendering dynamic scenes and recovering static surface geometry, we propose a template-free method to reconstruct surface geometry …
ETH Zurich（苏黎世联邦理工学院）；University of Amsterdam（阿姆斯特丹大学）；KU Leuven（鲁汶大学）
Monocular depth reconstruction of complex and dynamic scenes is a highly challenging problem. While for rigid scenes learning-based methods have been offering promising results even in unsupervised cases, there exists little to no literature addressing the same for dynamic and deformable scenes. In this work, we present an unsupervised monocular framework for dense depth estimation of dynamic scenes, which jointly reconstructs rigid and non-rigid parts without explicitly modelling the camera motion. Using dense correspondences, we derive a training objective that aims to opportunistically pres…
3D Gaussian Splatting against Moving Objects for High-Fidelity Street Scene Reconstruction
Ontario Knowledge Intelligence Centre, Canada（加拿大安大略知识智能中心）；Canadian Institute of Interscience, Canada（加拿大跨学科科学研究所）；DeepCox Intelligence, Canada（加拿大DeepCox智能科技公司）
This project presents an exploration into 3D scene reconstruction of synthetic and real-world scenes using Neural Radiance Field (NeRF) approaches. We primarily take advantage of the reduction in training and rendering time of neural graphic primitives multi-resolution hash encoding, to reconstruct static video game scenes and real-world scenes, comparing and observing reconstruction detail and limitations. Additionally, we explore dynamic scene reconstruction using Neural Radiance Fields for Dynamic Scenes(D-NeRF). Finally, we extend the implementation of D-NeRF, originally constrained to han…
School of Intelligence Science and Technology, Peking University（北京理工大学智能科学与技术学院）；National Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；AIR, Tsinghua University（清华大学人工智能研究院）
Crime Scene Investigation (CSI) is a carefully planned systematic process with the purpose of acquiring physical evidences to shed light upon the physical reality of the crime and eventually detect the identity of the criminal. Capturing images and videos of the crime scene is an important part of this process in order to conduct a deeper analysis on the digital evidence for possible hints. This work brings this idea further to use the acquired footage for generating a 3D model of the crime scene. Results show that realistic reconstructions can be obtained using sophisticated computer vision t…
Australian National University（澳大利亚国立大学）；Northwestern Polytechnical University（西北工业大学）；Australia Centre for Robotic Vision（澳大利亚机器人视觉中心）
This paper proposes a new approach for monocular dense 3D reconstruction of a complex dynamic scene from two perspective frames. By applying superpixel over-segmentation to the image, we model a generically dynamic (hence non-rigid) scene with a piecewise planar and rigid approximation. In this way, we reduce the dynamic reconstruction problem to a "3D jigsaw puzzle" problem which takes pieces from an unorganized "soup of superpixels". We show that our method provides an effective solution to the inherent relative scale ambiguity in structure-from-motion. Since our method does not assume a tem…
We present a novel approach to reconstruct RGB-D indoor scene with plane primitives. Our approach takes as input a RGB-D sequence and a dense coarse mesh reconstructed by some 3D reconstruction method on the sequence, and generate a lightweight, low-polygonal mesh with clear face textures and sharp features without losing geometry details from the original scene. To achieve this, we firstly partition the input mesh with plane primitives, simplify it into a lightweight mesh next, then optimize plane parameters, camera poses and texture colors to maximize the photometric consistency across frame…
Neural Radiance Fields (NeRF) has achieved impressive results in single object scene reconstruction and novel view synthesis, which have been demonstrated on many single modality and single object focused indoor scene datasets like DTU, BMVS, and NeRF Synthetic.However, the study of NeRF on large-scale outdoor scene reconstruction is still limited, as there is no unified outdoor scene dataset for large-scale NeRF evaluation due to expensive data acquisition and calibration costs. In this paper, we propose a large-scale outdoor multi-modal dataset, OMMO dataset, containing complex land objects …
Technical University of Munich（慕尼黑工业大学）；Google（谷歌）
We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion. Our self-supervised approach learns to jointly inpaint geometry and color by correlating an incomplete RGB-D scan with a more complete version of that scan. Notably, rather than relying on 3D reconstruction losses to inform our 3D geometry and color reconstruction, we propose adversarial and perceptual losses operating on 2D renderings in order to achieve high-resolution, high-quality colored …
