Implicit neural rendering techniques have shown promising results for novel view synthesis. However, existing methods usually encode the entire scene as a whole, which is generally not aware of the object identity and limits the ability to the high-level editing tasks such as moving or adding furniture. In this paper, we present a novel neural scene rendering system, which learns an object-compositional neural radiance field and produces realistic rendering with editing capability for a clustered and real-world scene. Specifically, we design a novel two-pathway architecture, in which the scene branch encodes the scene geometry and appearance, and the object branch encodes each standalone object conditioned on learnable object activation codes. To survive the training in heavily cluttered scenes, we propose a scene-guided training strategy to solve the 3D space ambiguity in the occluded regions and learn sharp boundaries for each object. Extensive experiments demonstrate that our system not only achieves competitive performance for static scene novel-view synthesis, but also produces realistic rendering for object-level editing.
Neural radiance fields (NeRF) achieve highly photo-realistic novel-view synthesis, but it's a challenging problem to edit the scenes modeled by NeRF-based methods, especially for dynamic scenes. We propose editable neural radiance fields that enable end-users to easily edit dynamic scenes and even support topological changes. Input with an image sequence from a single camera, our network is trained fully automatically and models topologically varying dynamics using our picked-out surface key points. Then end-users can edit the scene by easily dragging the key points to desired new positions. T…
Semantic 3D-aware Portrait Synthesis and Manipulation Based on Compositional Neural Radiance Field
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；ByteDance Ltd（字节跳动公司）；Nanjing University（南京大学）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Tencent AI Lab（腾讯人工智能实验室）；Tsinghua University（清华大学）
Previous portrait image generation methods roughly fall into two categories: 2D GANs and 3D-aware GANs. 2D GANs can generate high fidelity portraits but with low view consistency. 3D-aware GAN methods can maintain view consistency but their generated images are not locally editable. To overcome these limitations, we propose FENeRF, a 3D-aware generator that can produce view-consistent and locally-editable portrait images. Our method uses two decoupled latent codes to generate corresponding facial semantics and texture in a spatial aligned 3D volume with shared geometry. Benefiting from such un…
Generating Editable Head Avatars with 3D Gaussian GANs
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
TU Berlin（柏林工业大学）；UC San Diego（加州大学圣地亚哥分校）；MIT（麻省理工学院）
We present a method to learn compositional multi-object dynamics models from image observations based on implicit object encoders, Neural Radiance Fields (NeRFs), and graph neural networks. NeRFs have become a popular choice for representing scenes due to their strong 3D prior. However, most NeRF approaches are trained on a single scene, representing the whole scene with a global model, making generalization to novel scenes, containing different numbers of objects, challenging. Instead, we present a compositional, object-centric auto-encoder framework that maps multiple views of the scene to a…
Jagiellonian University, Faculty of Mathematics and Computer Science（杰洛尼摩大学数学与计算机科学学院）；Poznań University of Technology, Institute of Robotics and Machine Intelligence（波兹南技术大学机器人与人工智能研究所）；IDEAS Research Institute（IDEAS研究所）
University of Tübingen（图宾根大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Snap Inc.（Snap公司）
We present a novel method for performing flexible, 3D-aware image content manipulation while enabling high-quality novel view synthesis. While NeRF-based approaches are effective for novel view synthesis, such models memorize the radiance for every point in a scene within a neural network. Since these models are scene-specific and lack a 3D scene representation, classical editing such as shape manipulation, or combining scenes is not possible. Hence, editing and combining NeRF-based scenes has not been demonstrated. With the aim of obtaining interpretable and controllable scene representations…
Preferred Networks, Inc.（首选网络公司）；Massachusetts Institute of Technology（麻省理工学院）
Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations. However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional. In particular, it has been difficult to selectively edit specific regions or objects. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes. We…
3D generative models of objects enable photorealistic image synthesis with 3D control. Existing methods model the scene as a global scene representation, ignoring the compositional aspect of the scene. Compositional reasoning can enable a wide variety of editing applications, in addition to enabling generalizable 3D reasoning. In this paper, we present a compositional generative model, where each semantic part of the object is represented as an independent 3D representation learned from only in-the-wild 2D data. We start with a global generative model (GAN) and learn to decompose it into diffe…
Carnegie Mellon University（卡内基梅隆大学）；Facebook AI Research（Facebook人工智能研究院）；Facebook Reality Labs（Facebook现实实验室）
Photorealistic rendering of dynamic humans is an important ability for telepresence systems, virtual shopping, synthetic data generation, and more. Recently, neural rendering methods, which combine techniques from computer graphics and machine learning, have created high-fidelity models of humans and objects. Some of these methods do not produce results with high-enough fidelity for driveable human models (Neural Volumes) whereas others have extremely long rendering times (NeRF). We propose a novel compositional 3D representation that combines the best of previous methods to produce both highe…
Real2Sim: A Physics-driven and Editable Gaussian Splatting Framework for Autonomous Driving Scenes
Department of Civil and Environmental Engineering, Rensselaer Polytechnic Institute（拉特克利夫理工学院土木与环境工程系）；Department of Computer and Information Sciences, University of Delaware（德雷塞尔大学计算机与信息科学系）
ASSIST: Interactive Scene Nodes for Scalable and Realistic Indoor Simulation
Institute for AI Industry Research (AIR), Tsinghua University（清华大学智能产业研究院（AIR））；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Beijing University of Civil Engineering and Architecture（北京建筑大学）；University of Science and Technology of China（中国科学技术大学）；McGill University（麦吉尔大学）；Beijing Institute（北京研究所）
Stanford University（斯坦福大学）；TU Delft（代尔夫特理工大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Unsupervised learning of 3D-aware generative adversarial networks (GANs) using only collections of single-view 2D photographs has very recently made much progress. These 3D GANs, however, have not been demonstrated for human bodies and the generated radiance fields of existing frameworks are not directly editable, limiting their applicability in downstream tasks. We propose a solution to these challenges by developing a 3D GAN framework that learns to generate radiance fields of human bodies or faces in a canonical pose and warp them using an explicit deformation field into a desired body pose…
Car-Studio: Learning Car Radiance Fields from Single-View and Endless In-the-wild Images
The Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology(Guangzhou)（香港科技大学（广州））
ShanghaiTech University（上海科技大学）；Stereye Intelligent Technology Co.,Ltd.（视辰智能科技有限公司）；Huazhong University of Science and Technology（华中科技大学）
Some of the most exciting experiences that Metaverse promises to offer, for instance, live interactions with virtual characters in virtual environments, require real-time photo-realistic rendering. 3D reconstruction approaches to rendering, active or passive, still require extensive cleanup work to fix the meshes or point clouds. In this paper, we present a neural volumography technique called neural volumetric video or NeuVV to support immersive, interactive, and spatial-temporal rendering of volumetric video contents with photo-realism and in real-time. The core of NeuVV is to efficiently en…
ShanghaiTech University（上海科技大学）；Stereye Intelligent Technology Co.,Ltd.（眸视智能科技有限公司）；DGene Digital Technology Co., Ltd.（叠境数字科技有限公司）
Generating free-viewpoint videos is critical for immersive VR/AR experience but recent neural advances still lack the editing ability to manipulate the visual perception for large dynamic scenes. To fill this gap, in this paper we propose the first approach for editable photo-realistic free-viewpoint video generation for large-scale dynamic scenes using only sparse 16 cameras. The core of our approach is a new layered neural representation, where each dynamic entity including the environment itself is formulated into a space-time coherent neural layered radiance representation called ST-NeRF. …
VIRGi: View-dependent Instant Recoloring of 3D Gaussians Splats
Arquimea Research Center（阿基米德研究中心）；Universidad Politécnica de Catalunya（加泰罗尼亚理工大学）；Volinga AI；Universidad Politécnica de Madrid（马德里理工大学）；Universidad Rey Juan Carlos（雷伊·胡安·卡洛斯大学）；IUSANI, Universidad de Las Palmas de Gran Canaria（IUSANI，拉斯帕尔马斯德格兰坎亚大学）；Institut de Robòtica i Informàtica Industrial (IRI), CSIC-UPC（机器人与信息技术研究所（IRI），CSIC-UPC）
EAG-PT: Emission-Aware Gaussians and Path Tracing for Diffuse Indoor Scene Reconstruction and Editing
Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）
