NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo
Department of Automation, Tsinghua University（清华大学自动化系）；State Key Lab of Intelligent Technologies and Systems（智能技术与系统国家重点实验室）；ETH Zurich（苏黎世联邦理工学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, Jie Zhou
In this work, we present a new multi-view depth estimation method that utilizes both conventional reconstruction and learning-based priors over the recently proposed neural radiance fields (NeRF). Unlike existing neural network based optimization method that relies on estimated correspondences, our method directly optimizes over implicit volumes, eliminating the challenging step of matching pixels in indoor scenes. The key to our approach is to utilize the learning-based priors to guide the optimization process of NeRF. Our system firstly adapts a monocular depth network over the target scene by finetuning on its sparse SfM+MVS reconstruction from COLMAP. Then, we show that the shape-radiance ambiguity of NeRF still exists in indoor environments and propose to address the issue by employing the adapted depth priors to monitor the sampling process of volume rendering. Finally, a per-pixel confidence map acquired by error computation on the rendered image can be used to further improve the depth quality. Experiments show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes, with surprising findings presented on the effectiveness of correspondence-based optimization and NeRF-based optimization over the adapted depth priors. In addition, we show that the guided optimization scheme does not sacrifice the original synthesis capability of neural radiance fields, improving the rendering quality on both seen and novel views. Code is available at https://github.com/weiyithu/NerfingMVS.
ShanghaiTech University（上海科技大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Adobe Research（Adobe研究院）
We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis. Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference. Our approach leverages plane-swept cost volumes (widely used in multi-view stereo) for geometry-aware scene reasoning, and combines this with physically based volume rendering for neural radiance field reconstruction. W…
ETH Zürich（苏黎世联邦理工学院）；Google Research（谷歌研究院）；KU Leuven（鲁汶大学）
We present a modern solution to the multi-view photometric stereo problem (MVPS). Our work suitably exploits the image formation model in a MVPS experimental setup to recover the dense 3D reconstruction of an object from images. We procure the surface orientation using a photometric stereo (PS) image formation model and blend it with a multi-view neural radiance field representation to recover the object's surface geometry. Contrary to the previous multi-staged framework to MVPS, where the position, iso-depth contours, or orientation measurements are estimated independently and then fused late…
University of Tübingen（图宾根大学）；Max Planck Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克信息学研究所，萨尔兰信息学园区）；Carnegie Mellon University（卡内基梅隆大学）
Recent neural view synthesis methods have achieved impressive quality and realism, surpassing classical pipelines which rely on multi-view reconstruction. State-of-the-Art methods, such as NeRF, are designed to learn a single scene with a neural network and require dense multi-view inputs. Testing on a new scene requires re-training from scratch, which takes 2-3 days. In this work, we introduce Stereo Radiance Fields (SRF), a neural view synthesis approach that is trained end-to-end, generalizes to new scenes, and requires only sparse views at test time. The core idea is a neural architecture …
Multi-View Stereo (MVS) is a core task in 3D computer vision. With the surge of novel deep learning methods, learned MVS has surpassed the accuracy of classical approaches, but still relies on building a memory intensive dense cost volume. Novel View Synthesis (NVS) is a parallel line of research and has recently seen an increase in popularity with Neural Radiance Field (NeRF) models, which optimize a per scene radiance field. However, NeRF methods do not generalize to novel scenes and are slow to train and test. We propose to bridge the gap between these two methodologies with a novel network…
Nanyang Technological University（南洋理工大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Wenzhou University（温州大学）；UCAS-Terminus AI Lab, UCAS（中国科学院大学-特米纳斯人工智能实验室，中国科学院大学）
Shanghai Jiao Tong University（上海交通大学）；New York University（纽约大学）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；Peng Cheng Laboratory（鹏城实验室）
Virtual Reality (VR) is becoming ubiquitous with the rise of consumer displays and commercial VR platforms. Such displays require low latency and high quality rendering of synthetic imagery with reduced compute overheads. Recent advances in neural rendering showed promise of unlocking new possibilities in 3D computer graphics via image-based representations of virtual or physical environments. Specifically, the neural radiance fields (NeRF) demonstrated that photo-realistic quality and continuous view changes of 3D scenes can be achieved without loss of view-dependent effects. While NeRF can s…
Novel view synthesis (NVS) is a challenging task in computer vision that involves synthesizing new views of a scene from a limited set of input images. Neural Radiance Fields (NeRF) have emerged as a powerful approach to address this problem, but they require accurate knowledge of camera \textit{intrinsic} and \textit{extrinsic} parameters. Traditionally, structure-from-motion (SfM) and multi-view stereo (MVS) approaches have been used to extract camera parameters, but these methods can be unreliable and may fail in certain cases. In this paper, we propose a novel technique that leverages unpo…
University of Southern California（南加州大学）；Adobe Research（奥多比研究院）
Volumetric neural rendering methods like NeRF generate high-quality view synthesis results but are optimized per-scene leading to prohibitive reconstruction time. On the other hand, deep multi-view stereo methods can quickly reconstruct scene geometry via direct network inference. Point-NeRF combines the advantages of these two approaches by using neural 3D point clouds, with associated neural features, to model a radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Moreover, Point-NeRF can …
Reconstruction of the soft tissues in robotic surgery from endoscopic stereo videos is important for many applications such as intra-operative navigation and image-guided robotic surgery automation. Previous works on this task mainly rely on SLAM-based approaches, which struggle to handle complex surgical scenes. Inspired by recent progress in neural rendering, we present a novel framework for deformable tissue reconstruction from binocular captures in robotic surgery under the single-viewpoint setting. Our framework adopts dynamic neural radiance fields to represent deformable surgical scenes…
Thin, reflective objects such as forks and whisks are common in our daily lives, but they are particularly challenging for robot perception because it is hard to reconstruct them using commodity RGB-D cameras or multi-view stereo techniques. While traditional pipelines struggle with objects like these, Neural Radiance Fields (NeRFs) have recently been shown to be remarkably effective for performing view synthesis on objects with thin structures or reflective materials. In this paper we explore the use of NeRF as a new source of supervision for robust robot vision systems. In particular, we dem…
We propose to tackle the multiview photometric stereo problem using an extension of Neural Radiance Fields (NeRFs), conditioned on light source direction. The geometric part of our neural representation predicts surface normal direction, allowing us to reason about local surface reflectance. The appearance part of our neural representation is decomposed into a neural bidirectional reflectance function (BRDF), learnt as part of the fitting process, and a shadow prediction network (conditioned on light source direction) allowing us to model the apparent BRDF. This balance of learnt components wi…
The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Traditional multi-view photometric stereo (MVPS) methods are often composed of multiple disjoint stages, resulting in noticeable accumulated errors. In this paper, we present a neural inverse rendering method for MVPS based on implicit representation. Given multi-view images of a non-Lambertian object illuminated by multiple unknown directional lights, our method jointly estimates the geometry, materials, and lights. Our method first employs multi-light images to estimate per-view surface normal maps, which are used to regularize the normals derived from the neural radiance field. It then join…
UC-NeRF: Uncertainty-aware Conditional Neural Radiance Fields from Endoscopic Sparse Views
CUHK T Stone Robotics Institute（香港中文大学T Stone机器人研究所）；The Chinese University of Hong Kong（香港中文大学）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Tencent AI Lab（腾讯AI实验室）；Hong Kong Center for Logistics Robotics（香港物流机器人中心）
The Oxford Spires Dataset: Benchmarking Large-Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods
Oxford Robotics Inst., Dept. of Eng. Science, Univ. of Oxford, UK（牛津大学机器人研究所、工程科学系）；Group of Automation, Robotics and Computer Vision (AUROVA), University of Alicante, Spain（自动化、机器人与计算机视觉小组（AUROVA）、阿尔基兰特大学）
BathyFacto: Refraction-Aware Two-Media Neural Radiance Fields for Bathymetry
Department of Geodesy and Geoinformation, TU Wien（维也纳技术大学测绘与地理信息系）；Institute of Photogrammetry and Remote Sensing (IPF), Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院测绘与遥感研究所）；Unit of Geometry and Surveying, University of Innsbruck（因斯布鲁克大学几何与测绘单位）
RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction
IRIT, UMR CNRS 5505（法国国家科学研究中心IRIT联合研究单位）；Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC（诺曼底大学、卡昂大学、卡昂高等工程师学院、法国国家科学研究中心GREYC联合研究单位）；Balgrist Hospital, University of Zurich（苏黎世大学巴尔格里斯医院）；OR-X
High-fidelity 3D Reconstruction of Plants using Neural Radiance Field
College of Engineering, South China Agriculture University（华南农业大学工程学院）；Department of Mechanical and Aerospace Engineering, Monash University（莫纳什大学机械与航空航天工程系）
NeVStereo: A NeRF-Driven NVS-Stereo Architecture for High-Fidelity 3D Tasks
University of Washington（华盛顿大学）；University of Southern California（南加州大学）；CUHK-Shenzhen（香港中文大学（深圳））；USC Institute for Creative Technologies（南加州大学创意技术研究所）
Through capturing spatial and angular radiance distribution, light field cameras introduce new capabilities that are not possible with conventional cameras. So far in the light field imaging literature, the focus has been on the theory and applications of single light field capture. By combining multiple light fields, it is possible to obtain new capabilities and enhancements, and even exceed physical limitations, such as spatial resolution and aperture size of the imaging device. In this paper, we present an algorithm to register and stitch multiple light fields. We utilize the regularity of …
We introduce the Satellite Neural Radiance Field (Sat-NeRF), a new end-to-end model for learning multi-view satellite photogrammetry in the wild. Sat-NeRF combines some of the latest trends in neural rendering with native satellite camera models, represented by rational polynomial coefficient (RPC) functions. The proposed method renders new views and infers surface models of similar quality to those obtained with traditional state-of-the-art stereo pipelines. Multi-date images exhibit significant changes in appearance, mainly due to varying shadows and transient objects (cars, vegetation). Rob…
Towards Learning a Generalizable 3D Scene Representation from 2D Observations
University of Hamburg - Department of Informatics（汉堡大学信息学院）；ZAL Center of Applied Aeronautical Research（应用航空研究中心）；Hamburger Informatik Technologie-Center e.V. (HITeC)（汉堡信息科技中心（HITeC））
Depth Supervised Neural Surface Reconstruction from Airborne Imagery
nFrames ESRI；Institute for Photogrammetry and Geoinformatics, University of Stuttgart（测绘与地理信息研究所，斯图加特大学）；Geoinformatics, University of Stuttgart（地理信息学，斯图加特大学）
Sat-DN: Implicit Surface Reconstruction from Multi-View Satellite Images with Depth and Normal Supervision
School of Remote Sensing Information Engineering, Wuhan University（遥感信息工程学院，武汉大学）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（测绘遥感信息工程国家重点实验室，武汉大学）
From Transparent to Opaque: Rethinking Neural Implicit Surfaces with $α$-NeuS
Key Laboratory of System Software (CAS) and State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Department of Computer Science, Stony Brook University（石溪大学计算机科学系）；SenseTime Research（商汤科技研究院）
MVPGS: Excavating Multi-view Priors for Gaussian Splatting from Sparse Input Views
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；School of Computer Science, University of Birmingham（伯明翰大学计算机学院）；Peng Cheng Laboratory（鹏城实验室）
C-NERF: Representing Scene Changes as Directional Consistency Difference-based NeRF
College of Computer Science and Technology, Civil Aviation University of China（中国民航大学计算机科学与技术学院）；IHPC, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所）；CFAR, Agency for Science, Technology and Research（新加坡科技研究局CFAR研究所）
Geometry-Consistent 4D Gaussian Splatting for Sparse-Input Dynamic View Synthesis
The Hong Kong Polytechnic University, Hong Kong, China（香港理工大学）；Southern University of Science and Technology, Shenzhen, China（南方科技大学）；Indian Institute of Technology, Jodhpur, India（印度理工学院，朱道尔）；Institute of Cyberspace Technology, HKCT Institute of Higher Education, Hong Kong, China（网络空间技术研究所，香港高等教育学院）
We present a novel neural surface reconstruction method called NeuralRoom for reconstructing room-sized indoor scenes directly from a set of 2D images. Recently, implicit neural representations have become a promising way to reconstruct surfaces from multiview images due to their high-quality results and simplicity. However, implicit neural representations usually cannot reconstruct indoor scenes well because they suffer severe shape-radiance ambiguity. We assume that the indoor scene consists of texture-rich and flat texture-less regions. In texture-rich regions, the multiview stereo can obta…
Meituan UAV（美团无人机）；School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Dept. of Mechanical Engineering, Computer and Information Sciences, University of Delaware（德雷塞尔大学机械工程、计算机与信息科学系）
Multi-view Surface Reconstruction Using Normal and Reflectance Cues
University of Zurich（苏黎世大学）；Université de Toulouse（图卢兹大学）；CNRS, UNICAEN, ENSICAEN, Normandie Université（法国国家科学研究中心、UNICAEN、ENSICAEN、诺曼底大学）；FittingBox（FittingBox公司）；University of Copenhagen（哥本哈根大学）
