Recent work on Neural Radiance Fields (NeRF) showed how neural networks can be used to encode complex 3D environments that can be rendered photorealistically from novel viewpoints. Rendering these images is very computationally demanding and recent improvements are still a long way from enabling interactive rates, even on high-end hardware. Motivated by scenarios on mobile and mixed reality devices, we propose FastNeRF, the first NeRF-based system capable of rendering high fidelity photorealistic images at 200Hz on a high-end consumer GPU. The core of our method is a graphics-inspired factorization that allows for (i) compactly caching a deep radiance map at each position in space, (ii) efficiently querying that map using ray directions to estimate the pixel values in the rendered image. Extensive experiments show that the proposed method is 3000 times faster than the original NeRF algorithm and at least an order of magnitude faster than existing work on accelerating NeRF, while maintaining visual quality and extensibility.
Recent neural rendering approaches greatly improve image quality, reaching near photorealism. However, the underlying neural networks have high runtime, precluding telepresence and virtual reality applications that require high resolution at low latency. The sequential dependency of layers in deep networks makes their optimization difficult. We break this dependency by caching information from the previous frame to speed up the processing of the current one with an implicit warp. The warping with a shallow network reduces latency and the caching operations can further be parallelized to improv…
Alaya Studio, Shanda AI Research Tokyo（Alaya工作室，盛大人工智能研究院东京）；National Taiwan University（国立台湾大学）；The University of Tokyo（东京大学）；National Yang Ming Chiao Tung University（国立阳明交通大学）
MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold
South China University of Technology；Columbia University；Orange Team, Youku Moku-Lab, HUJING Digital Media \& Entertainment Group；Singapore Management University
Disentangled 4D Gaussian Splatting: Rendering High-Resolution Dynamic World at 343 FPS
Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning, Central China Normal University（人工智能与智能学习湖北省重点实验室，中国地质大学（武汉））；School of Computer Science, Central China Normal University（计算机科学学院，中国地质大学（武汉））；National Language Resources Monitoring and Research Center for Network Media, Central China Normal University（网络媒体语言资源监测与研究中心，中国地质大学（武汉））；College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（人工智能学院，南京航空航天大学）；School of Data Science, Lingnan University（数据科学学院，岭南大学）
Imperial College London（帝国理工学院）；Huawei Technologies（华为技术有限公司）
We present Dynamic Neural Portraits, a novel approach to the problem of full-head reenactment. Our method generates photo-realistic video portraits by explicitly controlling head pose, facial expressions and eye gaze. Our proposed architecture is different from existing methods that rely on GAN-based image-to-image translation networks for transforming renderings of 3D faces into photo-realistic images. Instead, we build our system upon a 2D coordinate-based MLP with controllable dynamics. Our intuition to adopt a 2D-based representation, as opposed to recent 3D NeRF-like systems, stems from t…
GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting
The Hong Kong University of Science and Technology（香港科技大学）；SenseTime Research（商汤科技研究院）；Beijing Institute of Technology（北京理工大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiaotong University（上海交通大学）
To the best of our knowledge, we first present a live system that generates personalized photorealistic talking-head animation only driven by audio signals at over 30 fps. Our system contains three stages. The first stage is a deep neural network that extracts deep audio features along with a manifold projection to project the features to the target person's speech space. In the second stage, we learn facial dynamics and motions from the projected audio features. The predicted motions include head poses and upper body motions, where the former is generated by an autoregressive probabilistic mo…
Carnegie Mellon University（卡内基梅隆大学）；Tokyo Institute of Technology（东京工业大学）
We present RePOSE, a fast iterative refinement method for 6D object pose estimation. Prior methods perform refinement by feeding zoomed-in input and rendered RGB images into a CNN and directly regressing an update of a refined pose. Their runtime is slow due to the computational cost of CNN, which is especially prominent in multiple-object pose refinement. To overcome this problem, RePOSE leverages image rendering for fast feature extraction using a 3D model with a learnable texture. We call this deep texture rendering, which uses a shallow multi-layer perceptron to directly regress a view-inv…
Frequency-Decomposed Avatar Representation for Varying Camera Distances
Università degli Studi di Genova（热那亚大学）；Istituto Italiano di Tecnologia (IIT)（意大利技术研究院）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
Triangle Splatting for Real-Time Radiance Field Rendering
University of Liège（列日大学）；KAUST（阿卜杜拉国王科技大学）；University of Oxford（牛津大学）；Simon Fraser University（西蒙弗雷泽大学）；University of Toronto（多伦多大学）；Google DeepMind（谷歌DeepMind）
University of California, San Diego（加州大学圣地亚哥分校）；Adobe Research（Adobe研究院）
While NeRF has shown great success for neural reconstruction and rendering, its limited MLP capacity and long per-scene optimization times make it challenging to model large-scale indoor scenes. In contrast, classical 3D reconstruction methods can handle large-scale scenes but do not produce realistic renderings. We propose NeRFusion, a method that combines the advantages of NeRF and TSDF-based fusion techniques to achieve efficient large-scale reconstruction and photo-realistic rendering. We process the input image sequence to predict per-frame local radiance fields via direct network inferen…
Depth-Guided Bundle Sampling for Efficient Generalizable Neural Radiance Field Reconstruction
Key Laboratory of Media Audio and Video (Communication University of China), Ministry of Education（媒体音频与视频重点实验室（中国传媒大学），教育部）；School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）
针对可泛化神经辐射场渲染计算量大、采样冗余的问题，提出深度引导的束采样策略，通过分组采样和自适应分配采样点，在DTU数据集上实现1.27 dB PSNR提升和47% FPS增加，并达到2倍渲染加速。
The Chinese University of Hong Kong（香港中文大学）；SmartMore（思谋科技）
Neural Radiance Fields (NeRF) has been wildly applied to various tasks for its high-quality representation of 3D scenes. It takes long per-scene training time and per-image testing time. In this paper, we present EfficientNeRF as an efficient NeRF-based method to represent 3D scene and synthesize novel-view images. Although several ways exist to accelerate the training or testing process, it is still difficult to much reduce time for both phases simultaneously. We analyze the density and weight distribution of the sampled points then propose valid and pivotal sampling at the coarse and fine st…
LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPS
Harvard University（哈佛大学）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；Johns Hopkins University（约翰霍普金斯大学）；MIT-IBM Watson AI Lab（MIT-IBM Watson人工智能实验室）；UMass Amherst（马萨诸塞大学阿姆赫斯特分校）
Hochschule Bonn-Rhein-Sieg（波恩-莱茵-锡格应用技术大学）；La Sapienza Università di Roma（罗马第一大学）
NeRF-based techniques fit wide and deep multi-layer perceptrons (MLPs) to a continuous radiance field that can be rendered from any unseen viewpoint. However, the lack of surface and normals definition and high rendering times limit their usage in typical computer graphics applications. Such limitations have recently been overcome separately, but solving them together remains an open problem. We present KiloNeuS, a neural representation reconstructing an implicit surface represented as a signed distance function (SDF) from multi-view images and enabling real-time rendering by partitioning the …
Spectrally Pruned Gaussian Fields with Neural Compensation
Tsinghua University（清华大学）；AIR, Tsinghua University（清华大学AIR研究院）；Imperial College London（帝国理工学院）；Beihang University（北京航空航天大学）；Beijing Institute of Technology（北京理工大学）；UCAS（中国科学院大学）；CUHK (SZ)（香港中文大学（深圳））；TeleAI（中国电信科技研究院）；Institute of AI (TeleAI)（中国电信人工智能研究院）
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
School of CS, Huazhong University of Science and Technology（华中科技大学计算机学院）；School of EIC, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Huawei Inc.（华为公司）
UC Berkeley（加州大学伯克利分校）；USC Institute for Creative Technologies（南加州大学创意技术研究所）；Pinscreen
We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects. Our method can render 800x800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs. We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects. Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree. In order to preserve view-dependent effects such as speculariti…
