NeRF is a popular model that efficiently represents 3D objects from 2D images. However, vanilla NeRF has some important limitations. NeRF must be trained on each object separately. The training time is long since we encode the object's shape and color in neural network weights. Moreover, NeRF does not generalize well to unseen data. In this paper, we present MultiPlaneNeRF -- a model that simultaneously solves the above problems. Our model works directly on 2D images. We project 3D points on 2D images to produce non-trainable representations. The projection step is not parametrized and a very shallow decoder can efficiently process the representation. Furthermore, we can train MultiPlaneNeRF on a large data set and force our implicit decoder to generalize across many objects. Consequently, we can only replace the 2D images (without additional training) to produce a NeRF representation of the new object. In the experimental section, we demonstrate that MultiPlaneNeRF achieves results comparable to state-of-the-art models for synthesizing new views and has generalization properties. Additionally, MultiPlane decoder can be used as a component in large generative models like GANs.
Approximating radiance fields with volumetric grids is one of promising directions for improving NeRF, represented by methods like Plenoxels and DVGO, which achieve super-fast training convergence and real-time rendering. However, these methods typically require a tremendous storage overhead, costing up to hundreds of megabytes of disk space and runtime memory for a single scene. We address this issue in this paper by introducing a simple yet effective framework, called vector quantized radiance fields (VQRF), for compressing these volume-grid-based radiance fields. We first present a robust a…
We present a learning framework for reconstructing neural scene representations from a small number of unconstrained tourist photos. Since each image contains transient occluders, decomposing the static and transient components is necessary to construct radiance fields with such in-the-wild photographs where existing methods require a lot of training data. We introduce SF-NeRF, aiming to disentangle those two components with only a few images given, which exploits semantic information without any supervision. The proposed method contains an occlusion filtering module that predicts the transien…
Adaptive Time-step Training for Enhancing Spike-Based Neural Radiance Fields
School of Information Science and Technology, ShanghaiTech University（信息科学与技术学院，上海科技大学）；Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）
Neural radiance fields (NeRF) have demonstrated the potential of coordinate-based neural representation (neural fields or implicit neural representation) in neural rendering. However, using a multi-layer perceptron (MLP) to represent a 3D scene or object requires enormous computational resources and time. There have been recent studies on how to reduce these computational inefficiencies by using additional data structures, such as grids or trees. Despite the promising performance, the explicit data structure necessitates a substantial amount of memory. In this work, we present a method to redu…
IReNe: Instant Recoloring of Neural Radiance Fields
Arquimea Research Center（阿克梅亚研究中心）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）；Universidad Rey Juan Carlos（胡安·卡洛斯国王大学）；Volinga AI（Volinga AI公司）；SEDDI（SEDDI机构）；Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学机器人与工业信息研究所）；Universidad de las Palmas de Gran Canaria（大加那利岛拉斯帕尔马斯大学）；IUSIANI（IUSIANI机构）；Universidad Politécnica de Madrid（马德里理工大学）
Quantum Implicit Neural Representations for Novel View Synthesis
Department of Mathematics, Computer Science and Physics, University of Udine（数学、计算机科学与物理系，乌迪内大学）；Department of Biology, University of Naples Federico II（生物学系，那不勒斯费德里科二世大学）；Max Planck Institute for Informatics, SIC, Saarbrücken, Germany（马克斯·普朗克信息研究所，萨尔布吕肯，德国）
Neural Radiance Field (NeRF) has widely received attention in Sparse-View Computed Tomography (SVCT) reconstruction tasks as a self-supervised deep learning framework. NeRF-based SVCT methods represent the desired CT image as a continuous function of spatial coordinates and train a Multi-Layer Perceptron (MLP) to learn the function by minimizing loss on the SV sinogram. Benefiting from the continuous representation provided by NeRF, the high-quality CT image can be reconstructed. However, existing NeRF-based SVCT methods strictly suppose there is completely no relative motion during the CT acq…
2S-UDF: A Novel Two-stage UDF Learning Method for Robust Non-watertight Model Reconstruction from Multi-view Images
State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所计算机科学国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
NVIDIA（英伟达）；University of Toronto（多伦多大学）；ROBLOX（罗布乐思）；University of Waterloo（滑铁卢大学）；Adobe Research（奥多比研究院）
Neural approximations of scalar and vector fields, such as signed distance functions and radiance fields, have emerged as accurate, high-quality representations. State-of-the-art results are obtained by conditioning a neural approximation with a lookup from trainable feature grids that take on part of the learning task and allow for smaller, more efficient neural networks. Unfortunately, these feature grids usually come at the cost of significantly increased memory consumption compared to stand-alone neural network models. We present a dictionary method for compressing such feature grids, redu…
