Neural Radiance Fields (NeRF) have quickly become the primary approach for 3D reconstruction and novel view synthesis in recent years due to their remarkable performance. Despite the huge interest in NeRF methods, a practical use case of NeRFs has largely been ignored; the exploration of the scene space modelled by a NeRF. In this paper, for the first time in the literature, we propose and formally define the scene exploration framework as the efficient discovery of NeRF model inputs (i.e. coordinates and viewing angles), using which one can render novel views that adhere to user-selected criteria. To remedy the lack of approaches addressing scene exploration, we first propose two baseline methods called Guided-Random Search (GRS) and Pose Interpolation-based Search (PIBS). We then cast scene exploration as an optimization problem, and propose the criteria-agnostic Evolution-Guided Pose Search (EGPS) for efficient exploration. We test all three approaches with various criteria (e.g. saliency maximization, image quality maximization, photo-composition quality improvement) and show that our EGPS performs more favourably than other baselines. We finally highlight key points and limitations, and outline directions for future research in scene exploration.
Neural Radiance Fields (NeRF) have attracted significant attention due to their ability to synthesize novel scene views with great accuracy. However, inherent to their underlying formulation, the sampling of points along a ray with zero width may result in ambiguous representations that lead to further rendering artifacts such as aliasing in the final scene. To address this issue, the recent variant mip-NeRF proposes an Integrated Positional Encoding (IPE) based on a conical view frustum. Although this is expressed with an integral formulation, mip-NeRF instead approximates this integral as th…
This project presents an exploration into 3D scene reconstruction of synthetic and real-world scenes using Neural Radiance Field (NeRF) approaches. We primarily take advantage of the reduction in training and rendering time of neural graphic primitives multi-resolution hash encoding, to reconstruct static video game scenes and real-world scenes, comparing and observing reconstruction detail and limitations. Additionally, we explore dynamic scene reconstruction using Neural Radiance Fields for Dynamic Scenes(D-NeRF). Finally, we extend the implementation of D-NeRF, originally constrained to han…
Neural volumetric representations have shown the potential that Multi-layer Perceptrons (MLPs) can be optimized with multi-view calibrated images to represent scene geometry and appearance, without explicit 3D supervision. Object segmentation can enrich many downstream applications based on the learned radiance field. However, introducing hand-crafted segmentation to define regions of interest in a complex real-world scene is non-trivial and expensive as it acquires per view annotation. This paper carries out the exploration of self-supervised learning for object segmentation using NeRF for co…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；ByteDance（字节跳动）；Hong Kong Polytechnic University（香港理工大学）；University of Maryland（马里兰大学）
Recent advances in neural rendering imply a future of widespread visual data distributions through sharing NeRF model weights. However, while common visual data (images and videos) have standard approaches to embed ownership or copyright information explicitly or subtly, the problem remains unexplored for the emerging NeRF format. We present StegaNeRF, a method for steganographic information embedding in NeRF renderings. We design an optimization framework allowing accurate hidden information extractions from images rendered by NeRF, while preserving its original visual quality. We perform exp…
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）
Children’s Hospital, Harvard Medical School（哈佛大学医学院儿童医院）；National University of Singapore（新加坡国立大学）；Agency for Science, Technology and Research (A*STAR)（科技研究局（A*STAR））；University of Minnesota Twin Cities（明尼苏达大学双城分校）
Searching for a target object in a cluttered scene constitutes a fundamental challenge in daily vision. Visual search must be selective enough to discriminate the target from distractors, invariant to changes in the appearance of the target, efficient to avoid exhaustive exploration of the image, and must generalize to locate novel target objects with zero-shot training. Previous work has focused on searching for perfect matches of a target after extensive category-specific training. Here we show for the first time that humans can efficiently and invariantly search for natural objects in compl…
Sapienza University of Rome（罗马第一大学）；Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）；University of California, San Diego（加州大学圣地亚哥分校）；OPAL AI INC.（OPAL人工智能公司）
The aim of this work is to introduce MaRF, a novel framework able to synthesize the Martian environment using several collections of images from rover cameras. The idea is to generate a 3D scene of Mars' surface to address key challenges in planetary surface exploration such as: planetary geology, simulated navigation and shape analysis. Although there exist different methods to enable a 3D reconstruction of Mars' surface, they rely on classical computer graphics techniques that incur high amounts of computational resources during the reconstruction process, and have limitations with generaliz…
GaussReg: Fast 3D Registration with Gaussian Splatting
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；The Future Network of Intelligence Institute, CUHK-Shenzhen（香港中文大学（深圳）未来智能网络研究院）
National University of Singapore（新加坡国立大学）；Tencent（腾讯）
Modeling dynamic scenes is important for many applications such as virtual reality and telepresence. Despite achieving unprecedented fidelity for novel view synthesis in dynamic scenes, existing methods based on Neural Radiance Fields (NeRF) suffer from slow convergence (i.e., model training time measured in days). In this paper, we present DeVRF, a novel representation to accelerate learning dynamic radiance fields. The core of DeVRF is to model both the 3D canonical space and 4D deformation field of a dynamic, non-rigid scene with explicit and discrete voxel-based representations. However, i…
GS-Hider: Hiding Messages into 3D Gaussian Splatting
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；Peking University Shenzhen Graduate School-Rabbitpre AIGC Joint Research Laboratory（北京大学深圳研究生院-Rabbitpre AIGC联合实验室）
StylizedGS: Controllable Stylization for 3D Gaussian Splatting
University of Chinese Academy of Sciences（中国科学院大学）；Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences（北京移动计算与 pervasive 设备重点实验室，中国科学院计算技术研究所）；Victoria University of Wellington（维多利亚大学）；Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences（智能信息处理重点实验室，中国科学院计算技术研究所）
