While neural radiance fields (NeRF) led to a breakthrough in photorealistic novel view synthesis, handling mirroring surfaces still denotes a particular challenge as they introduce severe inconsistencies in the scene representation. Previous attempts either focus on reconstructing single reflective objects or rely on strong supervision guidance in terms of additional user-provided annotations of visible image regions of the mirrors, thereby limiting the practical usability. In contrast, in this paper, we present NeRF-MD, a method which shows that NeRFs can be considered as mirror detectors and which is capable of reconstructing neural radiance fields of scenes containing mirroring surfaces without the need for prior annotations. To this end, we first compute an initial estimate of the scene geometry by training a standard NeRF using a depth reprojection loss. Our key insight lies in the fact that parts of the scene corresponding to a mirroring surface will still exhibit a significant photometric inconsistency, whereas the remaining parts are already reconstructed in a plausible manner. This allows us to detect mirror surfaces by fitting geometric primitives to such inconsistent regions in this initial stage of the training. Using this information, we then jointly optimize the radiance field and mirror geometry in a second training stage to refine their quality. We demonstrate the capability of our method to allow the faithful detection of mirrors in the scene as well as the reconstruction of a single consistent scene representation, and demonstrate its potential in comparison to baseline and mirror-aware approaches.
Universität Hamburg（汉堡大学）；Deutsches Elektronen-Synchrotron DESY（德国电子同步加速器研究所（DESY））
Neural radiance field (NeRF), in particular its extension by instant neural graphics primitives, is a novel rendering method for view synthesis that uses real-world images to build photo-realistic immersive virtual scenes. Despite its potential, research on the combination of NeRF and virtual reality (VR) remains sparse. Currently, there is no integration into typical VR systems available, and the performance and suitability of NeRF implementations for VR have not been evaluated, for instance, for different scene complexities or screen resolutions. In this paper, we present and evaluate a NeRF…
Zhejiang University（浙江大学）；Singapore University of Technology and Design（新加坡科技设计大学）；University of Tübingen（图宾根大学）；MPI-IS（马克斯·普朗克智能系统研究所）
Large-scale training data with high-quality annotations is critical for training semantic and instance segmentation models. Unfortunately, pixel-wise annotation is labor-intensive and costly, raising the demand for more efficient labeling strategies. In this work, we present a novel 3D-to-2D label transfer method, Panoptic NeRF, which aims for obtaining per-pixel 2D semantic and instance labels from easy-to-obtain coarse 3D bounding primitives. Our method utilizes NeRF as a differentiable tool to unify coarse 3D annotations and 2D semantic cues transferred from existing datasets. We demonstrat…
DenseSplat: Densifying Gaussian Splatting SLAM with Neural Radiance Prior
Department of Computer Science, Dalian University of Technology（大连理工大学计算机科学系）；Department of Mechano-informatics, Information Science and Technology, The University of Tokyo（东京大学机械信息学系、信息科学与技术系）；Institute of Medical Robotics and Department of Automation, Shanghai Jiao Tong University（上海交通大学医学机器人研究所、自动化系）；Key Laboratory of System Control and Information Processing, Ministry of Education（教育部系统控制与信息处理重点实验室）
Tsinghua University（清华大学）；Tencent AI Lab（腾讯人工智能实验室）；BIMSA（北京国际数学研究中心）
Neural Radiance Fields (NeRF) has achieved unprecedented view synthesis quality using coordinate-based neural scene representations. However, NeRF's view dependency can only handle simple reflections like highlights but cannot deal with complex reflections such as those from glass and mirrors. In these scenarios, NeRF models the virtual image as real geometries which leads to inaccurate depth estimation, and produces blurry renderings when the multi-view consistency is violated as the reflected objects may only be seen under some of the viewpoints. To overcome these issues, we introduce NeRFRe…
This project presents an exploration into 3D scene reconstruction of synthetic and real-world scenes using Neural Radiance Field (NeRF) approaches. We primarily take advantage of the reduction in training and rendering time of neural graphic primitives multi-resolution hash encoding, to reconstruct static video game scenes and real-world scenes, comparing and observing reconstruction detail and limitations. Additionally, we explore dynamic scene reconstruction using Neural Radiance Fields for Dynamic Scenes(D-NeRF). Finally, we extend the implementation of D-NeRF, originally constrained to han…
Seeing Through Reflections: Advancing 3D Scene Reconstruction in Mirror-Containing Environments with Gaussian Splatting
School of Automation and Intelligent Sening, Shanghai Jiao Tong University（自动化与智能传感学院，上海交通大学）；Ningbo Artificial Intelligence Institute, Shanghai Jiao Tong University（宁波人工智能研究所，上海交通大学）
SCIGS: 3D Gaussians Splatting from a Snapshot Compressive Image
National Engineering Research Center for Visual Information and Applications（视觉信息与应用国家工程研究中心）；School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）
NVIDIA（英伟达）；Georgia Institute of Technology（佐治亚理工学院）
We present a parallelized optimization method based on fast Neural Radiance Fields (NeRF) for estimating 6-DoF pose of a camera with respect to an object or scene. Given a single observed RGB image of the target, we can predict the translation and rotation of the camera by minimizing the residual between pixels rendered from a fast NeRF model and pixels in the observed image. We integrate a momentum-based camera extrinsic optimization procedure into Instant Neural Graphics Primitives, a recent exceptionally fast NeRF implementation. By introducing parallel Monte Carlo sampling into the pose es…
Triangle Splatting for Real-Time Radiance Field Rendering
University of Liège（列日大学）；KAUST（阿卜杜拉国王科技大学）；University of Oxford（牛津大学）；Simon Fraser University（西蒙弗雷泽大学）；University of Toronto（多伦多大学）；Google DeepMind（谷歌DeepMind）
Spectrally Pruned Gaussian Fields with Neural Compensation
Tsinghua University（清华大学）；AIR, Tsinghua University（清华大学AIR研究院）；Imperial College London（帝国理工学院）；Beihang University（北京航空航天大学）；Beijing Institute of Technology（北京理工大学）；UCAS（中国科学院大学）；CUHK (SZ)（香港中文大学（深圳））；TeleAI（中国电信科技研究院）；Institute of AI (TeleAI)（中国电信人工智能研究院）
UGOD: Uncertainty-Guided Opacity and Dropout for Sparse-View 3D Gaussian Splatting
Manchester Metropolitan University（曼彻斯特城市大学）；University of Surrey（萨里大学）；Imperial College London（帝国理工学院）；University of Exeter（埃克塞特大学）；Southeast University（东南大学）；Nanjing University of Science and Technology（南京理工大学）；University of Leeds（利兹大学）；Aston University（阿斯顿大学）
A Survey on 3D Reconstruction Techniques in Plant Phenotyping: From Classical Methods to Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and Beyond
Department of Electrical and Computer Engineering, Michigan State University（电气与计算机工程系，密歇根州立大学）；Department of Systems and Information Engineering & Link Lab, University of Virginia（系统与信息工程系及Link实验室，弗吉尼亚大学）；Department of Statistics and Probability, Michigan State University（统计与概率系，密歇根州立大学）；School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University（化学、化工与生物技术学院，南洋理工大学）；Department of Agricultural and Biological Engineering, Mississippi State University（农业与生物工程系，密西西比州立大学）；Department of Mechanical Engineering, Michigan State University（机械工程系，密歇根州立大学）
MeshSplatting: Differentiable Rendering with Opaque Meshes
University of Liège（里耶克斯大学）；Simon Fraser University（西蒙弗雷泽大学）；University of Maryland（马里兰大学）；University of British Columbia（不列颠哥伦比亚大学）；University of Toronto（多伦多大学）；Adobe Research（Adobe研究院）
Accurate and Complete Surface Reconstruction from 3D Gaussians via Direct SDF Learning
Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University（航空工程系，香港理工大学）；Department of Computer Science and Technology, Nanjing University（计算机科学与技术系，南京大学）；School of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（人工智能学院，南京航空航天大学）
Triangle Splatting+: Differentiable Rendering with Opaque Triangles
University of Liège（利根大学）；Simon Fraser University（西蒙弗雷泽大学）；University of Maryland（马里兰大学）；University of British Columbia（不列颠哥伦比亚大学）；University of Toronto（多伦多大学）；Adobe Research（Adobe研究）
Opacity-Gradient Driven Density Control for Compact and Efficient Few-Shot 3D Gaussian Splatting
Department of Computer Science and Physics, Wilfrid Laurier University（计算机科学与物理系，威尔弗里德·劳里尔大学）；Department of Computer Science（计算机科学系）；Physics, Wilfrid Laurier University（物理系，威尔弗里德·劳里尔大学）
