In this work, we propose the use of Neural Radiance Fields (NeRF) as a scene representation for visual localization. Recently, NeRF has been employed to enhance pose regression and scene coordinate regression models by augmenting the training database, providing auxiliary supervision through rendered images, or serving as an iterative refinement module. We extend its recognized advantages -- its ability to provide a compact scene representation with realistic appearances and accurate geometry -- by exploring the potential of NeRF's internal features in establishing precise 2D-3D matches for localization. To this end, we conduct a comprehensive examination of NeRF's implicit knowledge, acquired through view synthesis, for matching under various conditions. This includes exploring different matching network architectures, extracting encoder features at multiple layers, and varying training configurations. Significantly, we introduce NeRFMatch, an advanced 2D-3D matching function that capitalizes on the internal knowledge of NeRF learned via view synthesis. Our evaluation of NeRFMatch on standard localization benchmarks, within a structure-based pipeline, sets a new state-of-the-art for localization performance on Cambridge Landmarks.
Exploring Accurate 3D Phenotyping in Greenhouse through Neural Radiance Fields
Guangdong Academic of Agriculture Science（广东省农业科学院）；College of Engineering, South China Agriculture University（华南农业大学工程学院）；Department of Mechanical and Aerospace Engineering, Monash University（莫纳什大学机械与航空航天工程学院）
Nanyang Technological University（南洋理工大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Alibaba Group（阿里巴巴集团）；ByteDance（字节跳动）；Wenzhou University（温州大学）
Marrying NeRF with Feature Matching for One-step Pose Estimation
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；College of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google Research（谷歌研究院）；The Chinese University of Hong Kong（香港中文大学）
Neural Radiance Fields (NeRFs) are a powerful representation for modeling a 3D scene as a continuous function. Though NeRF is able to render complex 3D scenes with view-dependent effects, few efforts have been devoted to exploring its limits in a high-resolution setting. Specifically, existing NeRF-based methods face several limitations when reconstructing high-resolution real scenes, including a very large number of parameters, misaligned input data, and overly smooth details. In this work, we conduct the first pilot study on training NeRF with high-resolution data and propose the correspondi…
PlantSegNeRF: A few-shot, cross-species method for plant 3D instance point cloud reconstruction via joint-channel NeRF with multi-view image instance matching
The rendering procedure used by neural radiance fields (NeRF) samples a scene with a single ray per pixel and may therefore produce renderings that are excessively blurred or aliased when training or testing images observe scene content at different resolutions. The straightforward solution of supersampling by rendering with multiple rays per pixel is impractical for NeRF, because rendering each ray requires querying a multilayer perceptron hundreds of times. Our solution, which we call "mip-NeRF" (a la "mipmap"), extends NeRF to represent the scene at a continuously-valued scale. By efficient…
University of Cambridge, UK（剑桥大学）；China University of Mining and Technology（中国矿业大学）；ETH Zurich（苏黎世联邦理工学院）；University of California, Berkeley, USA（加州大学伯克利分校）；Shanghai Jiao Tong University（上海交通大学）
Department of Automation, Tsinghua University（清华大学自动化系）；State Key Lab of Intelligent Technologies and Systems（智能技术与系统国家重点实验室）；ETH Zurich（苏黎世联邦理工学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
In this work, we present a new multi-view depth estimation method that utilizes both conventional reconstruction and learning-based priors over the recently proposed neural radiance fields (NeRF). Unlike existing neural network based optimization method that relies on estimated correspondences, our method directly optimizes over implicit volumes, eliminating the challenging step of matching pixels in indoor scenes. The key to our approach is to utilize the learning-based priors to guide the optimization process of NeRF. Our system firstly adapts a monocular depth network over the target scene …
NeRF-Texture: Synthesizing Neural Radiance Field Textures
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；VAST；ARC Lab, Tencent PCG（腾讯PCG ARC实验室）；School of Computer Science & Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）
Neural Radiance Fields (NeRF) have attracted significant attention due to their ability to synthesize novel scene views with great accuracy. However, inherent to their underlying formulation, the sampling of points along a ray with zero width may result in ambiguous representations that lead to further rendering artifacts such as aliasing in the final scene. To address this issue, the recent variant mip-NeRF proposes an Integrated Positional Encoding (IPE) based on a conical view frustum. Although this is expressed with an integral formulation, mip-NeRF instead approximates this integral as th…
University at Buffalo（布法罗大学）；ETH Zürich（苏黎世联邦理工学院）；OPPO US Research Center（OPPO美国研究中心）；InnoPeak Tech（InnoPeak科技）；University of Tübingen（图宾根大学）
Visually exploring in a real-world 4D spatiotemporal space freely in VR has been a long-term quest. The task is especially appealing when only a few or even single RGB cameras are used for capturing the dynamic scene. To this end, we present an efficient framework capable of fast reconstruction, compact modeling, and streamable rendering. First, we propose to decompose the 4D spatiotemporal space according to temporal characteristics. Points in the 4D space are associated with probabilities of belonging to three categories: static, deforming, and new areas. Each area is represented and regular…
City University of Hong Kong（香港城市大学）；Snap Inc.（Snap公司）；USC Institute for Creative Technologies（南加州大学创意技术研究所）；Microsoft Cloud AI（微软云人工智能部门）
We present CLIP-NeRF, a multi-modal 3D object manipulation method for neural radiance fields (NeRF). By leveraging the joint language-image embedding space of the recent Contrastive Language-Image Pre-Training (CLIP) model, we propose a unified framework that allows manipulating NeRF in a user-friendly way, using either a short text prompt or an exemplar image. Specifically, to combine the novel view synthesis capability of NeRF and the controllable manipulation ability of latent representations from generative models, we introduce a disentangled conditional NeRF architecture that allows indiv…
Evaluating Modern Approaches in 3D Scene Reconstruction: NeRF vs Gaussian-Based Methods
Saarland University of Applied Science（萨尔应用科学大学）；Guangxi University（广西大学）；University of Florida（佛罗里达大学）；Columbia University（哥伦比亚大学）；Cornell University（康奈尔大学）；University of Michigan（密歇根大学）；Northeastern University（东北大学）；Harvard University（哈佛大学）；New York University（纽约大学）
Key Laboratory of Image Processing and Intelligent Control, Ministry of Education（信息处理与智能控制重点实验室）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（人工智能与自动化学院）；Nanyang Technological University (NTU)（南洋理工大学）
National Tsing Hua University（台湾清华大学）；ASUS AICS（华硕人工智能云端事业部）；Joint Research Center for AI Technology and All Vista Healthcare（人工智能技术与全瞻医疗联合研究中心）；Aeolus Robotics（Aeolus机器人公司）
We present a super-fast convergence approach to reconstructing the per-scene radiance field from a set of images that capture the scene with known poses. This task, which is often applied to novel view synthesis, is recently revolutionized by Neural Radiance Field (NeRF) for its state-of-the-art quality and flexibility. However, NeRF and its variants require a lengthy training time ranging from hours to days for a single scene. In contrast, our approach achieves NeRF-comparable quality and converges rapidly from scratch in less than 15 minutes with a single GPU. We adopt a representation consi…
We present an algorithm for generating novel views at arbitrary viewpoints and any input time step given a monocular video of a dynamic scene. Our work builds upon recent advances in neural implicit representation and uses continuous and differentiable functions for modeling the time-varying structure and the appearance of the scene. We jointly train a time-invariant static NeRF and a time-varying dynamic NeRF, and learn how to blend the results in an unsupervised manner. However, learning this implicit function from a single video is highly ill-posed (with infinitely many solutions that match…
Recent breakthroughs in text-guided image generation have led to remarkable progress in the field of 3D synthesis from text. By optimizing neural radiance fields (NeRF) directly from text, recent methods are able to produce remarkable results. Yet, these methods are limited in their control of each object's placement or appearance, as they represent the scene as a whole. This can be a major issue in scenarios that require refining or manipulating objects in the scene. To remedy this deficit, we propose a novel GlobalLocal training framework for synthesizing a 3D scene using object proxies. A p…
NeVStereo: A NeRF-Driven NVS-Stereo Architecture for High-Fidelity 3D Tasks
University of Washington（华盛顿大学）；University of Southern California（南加州大学）；CUHK-Shenzhen（香港中文大学（深圳））；USC Institute for Creative Technologies（南加州大学创意技术研究所）
IFFNeRF: Initialisation Free and Fast 6DoF pose estimation from a single image and a NeRF model
Fondazione Istituto Italiano di Tecnologia (IIT)（意大利技术研究院基金会）；Fondazione Bruno Kessler (FBK)（布鲁诺·凯塞勒基金会）；University of Trento（特伦托大学）；Durham University（杜伦大学）
Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（捷克布拉格技术大学捷克信息学、机器人学与控制论研究所）；ETH Zurich（苏黎世联邦理工学院）；Google DeepMind（谷歌DeepMind）；Faculty of Electrical Engineering, Czech Technical University in Prague（捷克布拉格技术大学电气工程学院）
