We introduce 3DEgo to address a novel problem of directly synthesizing photorealistic 3D scenes from monocular videos guided by textual prompts. Conventional methods construct a text-conditioned 3D scene through a three-stage process, involving pose estimation using Structure-from-Motion (SfM) libraries like COLMAP, initializing the 3D model with unedited images, and iteratively updating the dataset with edited images to achieve a 3D scene with text fidelity. Our framework streamlines the conventional multi-stage 3D editing process into a single-stage workflow by overcoming the reliance on COLMAP and eliminating the cost of model initialization. We apply a diffusion model to edit video frames prior to 3D scene creation by incorporating our designed noise blender module for enhancing multi-view editing consistency, a step that does not require additional training or fine-tuning of T2I diffusion models. 3DEgo utilizes 3D Gaussian Splatting to create 3D scenes from the multi-view consistent edited frames, capitalizing on the inherent temporal continuity and explicit point cloud data. 3DEgo demonstrates remarkable editing precision, speed, and adaptability across a variety of video sources, as validated by extensive evaluations on six datasets, including our own prepared GS25 dataset. Project Page: https://3dego.github.io/
3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting
The Chinese University of Hong Kong（香港中文大学）；Stanford University（斯坦福大学）；Snap Inc.（Snap公司）；University of California Los Angeles（加利福尼亚大学洛杉矶分校）；ByteDance（字节跳动）
Chat-Edit-3D++: Interactive 3D and 4D Scene Editing via Large Language Models
School of Electrical and Information Engineering, Beihang University（北京航空航天大学电子信息工程学院）；Institute of Unmanned System, Beihang University（北京航空航天大学无人系统研究院）；Atmanity Inc.（Atmanity公司）；Megvii Research（旷视研究院）；University of California at Merced（加州大学默塞德分校）
Neural radiance fields (NeRF) achieve highly photo-realistic novel-view synthesis, but it's a challenging problem to edit the scenes modeled by NeRF-based methods, especially for dynamic scenes. We propose editable neural radiance fields that enable end-users to easily edit dynamic scenes and even support topological changes. Input with an image sequence from a single camera, our network is trained fully automatically and models topologically varying dynamics using our picked-out surface key points. Then end-users can edit the scene by easily dragging the key points to desired new positions. T…
D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup
Doctoral School of Exact and Natural Sciences（精确与自然科学博士学院）；Jagiellonian University（雅盖隆大学）；Warsaw University of Technology（华沙工业大学）；Sano Centre for Computational Medicine（萨诺计算医学中心）；University of Cambridge（剑桥大学）；IDEAS NCBR
3DSceneEditor: Controllable 3D Scene Editing with Gaussian Splatting
D Optical Metrology unit, Fondazione Bruno Kessler（3D光学测绘单元，布鲁诺·凯斯勒基金会）；Department of Information Engineering and Computer Science, University of Trento（信息工程与计算机科学系，特伦托大学）
Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Meitu Inc.（美图公司）；Sun Yat-sen University（中山大学）；Beihang University（北京航空航天大学）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Computer and Control Engineering, University of Chinese Academy of Sciences（中国科学院大学计算机与控制工程学院）；School of Computer Science & Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；Alibaba Group（阿里巴巴集团）
Implicit neural rendering, especially Neural Radiance Field (NeRF), has shown great potential in novel view synthesis of a scene. However, current NeRF-based methods cannot enable users to perform user-controlled shape deformation in the scene. While existing works have proposed some approaches to modify the radiance field according to the user's constraints, the modification is limited to color editing or object translation and rotation. In this paper, we propose a method that allows users to perform controllable shape deformation on the implicit representation of the scene, and synthesizes t…
SVGS: Single-View to 3D Object Editing via Gaussian Splatting
School of Computer Science and Technology, Zhejiang Gongshang University（浙江工商大学计算机科学与技术学院）；Zhejiang Key Laboratory of Big Data and Future E-Commerce Technology（浙江省大数据与未来电子商务技术重点实验室）；Jianpei Technology Co., Ltd.（建培科技有限公司）；School of Artificial Intelligence and Computer Science, Nantong University（南通大学人工智能与计算机科学学院）；Department of Computer Science, College of Computer Engineering and Sciences, Prince Sattam Bin Abdulaziz University（沙特本·阿卜杜勒阿齐兹王子大学计算机工程与科学学院计算机科学系）；Department of Computer Science, Faculty of Computers and Information, Qena University（基纳大学计算机与信息学院计算机科学系）；Department of Computing Sciences, Faculty of Information Technology and Communication Sciences, Tampere University（坦佩雷大学信息技术与通信科学学院计算科学系）；Division of Oral and Maxillofacial Surgery, Faculty of Dentistry, The University of Hong Kong（香港大学牙医学院口腔颌面外科）；Systems Research Institute of the Polish Academy of Sciences（波兰科学院系统研究所）；AGH University of Krakow（AGH科技大学）；SAN University（SAN大学）
University of Tübingen（图宾根大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Snap Inc.（Snap公司）
We present a novel method for performing flexible, 3D-aware image content manipulation while enabling high-quality novel view synthesis. While NeRF-based approaches are effective for novel view synthesis, such models memorize the radiance for every point in a scene within a neural network. Since these models are scene-specific and lack a 3D scene representation, classical editing such as shape manipulation, or combining scenes is not possible. Hence, editing and combining NeRF-based scenes has not been demonstrated. With the aim of obtaining interpretable and controllable scene representations…
State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）；Google（谷歌）；The Chinese University of Hong Kong（香港中文大学）
Implicit neural rendering techniques have shown promising results for novel view synthesis. However, existing methods usually encode the entire scene as a whole, which is generally not aware of the object identity and limits the ability to the high-level editing tasks such as moving or adding furniture. In this paper, we present a novel neural scene rendering system, which learns an object-compositional neural radiance field and produces realistic rendering with editing capability for a clustered and real-world scene. Specifically, we design a novel two-pathway architecture, in which the scene…
Neural Radiance Field (NeRF) is a powerful tool to faithfully generate novel views for scenes with only sparse captured images. Despite its strong capability for representing 3D scenes and their appearance, its editing ability is very limited. In this paper, we propose a simple but effective extension of vanilla NeRF, named PaletteNeRF, to enable efficient color editing on NeRF-represented scenes. Motivated by recent palette-based image decomposition works, we approximate each pixel color as a sum of palette colors modulated by additive weights. Instead of predicting pixel colors as in vanilla…
In this work, we present I$^2$-SDF, a new method for intrinsic indoor scene reconstruction and editing using differentiable Monte Carlo raytracing on neural signed distance fields (SDFs). Our holistic neural SDF-based framework jointly recovers the underlying shapes, incident radiance and materials from multi-view images. We introduce a novel bubble loss for fine-grained small objects and error-guided adaptive sampling scheme to largely improve the reconstruction quality on large-scale indoor scenes. Further, we propose to decompose the neural radiance field into spatially-varying material of …
We present NeRFEditor, an efficient learning framework for 3D scene editing, which takes a video captured over 360° as input and outputs a high-quality, identity-preserving stylized 3D scene. Our method supports diverse types of editing such as guided by reference images, text prompts, and user interactions. We achieve this by encouraging a pre-trained StyleGAN model and a NeRF model to learn from each other mutually. Specifically, we use a NeRF model to generate numerous image-angle pairs to train an adjustor, which can adjust the StyleGAN latent code to generate high-fidelity stylized images…
GaussEdit: Adaptive 3D Scene Editing with Text and Image Prompts
School of Computer and Data Engineering, NingboTech University（计算机与数据工程学院，宁波科技学院）；Ningbo Institute, Zhejiang University（浙江大学宁波学院）；School of Software Technology, Zhejiang University（软件技术学院，浙江大学）；School of Big Data and Artificial Intelligence Management, Xi’an Jiaotong University（大数据与人工智能管理学院，西安交通大学）；School of Computer Science and Technology, ShanDong University（计算机科学与技术学院，山东大学）；Graphics & Geometric Computing Laboratory, School of Mathematical Sciences, University of Science and Technology of China（图形与几何计算实验室，数学科学学院，中国科学技术大学）
Preferred Networks, Inc.（首选网络公司）；Massachusetts Institute of Technology（麻省理工学院）
Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations. However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional. In particular, it has been difficult to selectively edit specific regions or objects. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes. We…
Recent advances in neural radiance fields have enabled the high-fidelity 3D reconstruction of complex scenes for novel view synthesis. However, it remains underexplored how the appearance of such representations can be efficiently edited while maintaining photorealism. In this work, we present PaletteNeRF, a novel method for photorealistic appearance editing of neural radiance fields (NeRF) based on 3D color decomposition. Our method decomposes the appearance of each 3D point into a linear combination of palette-based bases (i.e., 3D segmentations defined by a group of NeRF-type functions) tha…
MPI for Informatics（马克斯·普朗克信息学研究所）；Saarland University（萨尔大学）；University of York（约克大学）
Photorealistic editing of outdoor scenes from photographs requires a profound understanding of the image formation process and an accurate estimation of the scene geometry, reflectance and illumination. A delicate manipulation of the lighting can then be performed while keeping the scene albedo and geometry unaltered. We present NeRF-OSR, i.e., the first approach for outdoor scene relighting based on neural radiance fields. In contrast to the prior art, our technique allows simultaneous editing of both scene illumination and camera viewpoint using only a collection of outdoor photos shot in un…
