LatentEditor: Text Driven Local Editing of 3D Scenes
Center for Research in Computer Vision, University of Central Florida（中佛罗里达大学计算机视觉研究中心）；Department of Computer Science, Wayne State University（韦恩州立大学计算机科学系）
Umar Khalid, Hasan Iqbal, Nazmul Karim, Jing Hua, Chen Chen
While neural fields have made significant strides in view synthesis and scene reconstruction, editing them poses a formidable challenge due to their implicit encoding of geometry and texture information from multi-view inputs. In this paper, we introduce \textsc{LatentEditor}, an innovative framework designed to empower users with the ability to perform precise and locally controlled editing of neural fields using text prompts. Leveraging denoising diffusion models, we successfully embed real-world scenes into the latent space, resulting in a faster and more adaptable NeRF backbone for editing compared to traditional methods. To enhance editing precision, we introduce a delta score to calculate the 2D mask in the latent space that serves as a guide for local modifications while preserving irrelevant regions. Our novel pixel-level scoring approach harnesses the power of InstructPix2Pix (IP2P) to discern the disparity between IP2P conditional and unconditional noise predictions in the latent space. The edited latents conditioned on the 2D masks are then iteratively updated in the training set to achieve 3D local editing. Our approach achieves faster editing speeds and superior output quality compared to existing 3D editing models, bridging the gap between textual instructions and high-quality 3D scene editing in latent space. We show the superiority of our approach on four benchmark 3D datasets, LLFF, IN2N, NeRFStudio and NeRF-Art. Project Page: https://latenteditor.github.io/
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Saarland Informatics Campus（萨尔兰信息学园区）
While great progress has been made recently in automatic image manipulation, it has been limited to object centric images like faces or structured scene datasets. In this work, we take a step towards general scene-level image editing by developing an automatic interaction-free object removal model. Our model learns to find and remove objects from general scene images using image-level labels and unpaired data in a generative adversarial network (GAN) framework. We achieve this with two key contributions: a two-stage editor architecture consisting of a mask generator and image in-painter that c…
SceneCraft: Interactive System for Image Editing via Scene Graph
University of Science, Ho Chi Minh, Vietnam（胡志明市理科大学）；Vietnam National University, Ho Chi Minh, Vietnam（越南国家大学胡志明市）；University of Dayton, Dayton, Ohio, USA（代顿大学）
Catalyst4D: High-Fidelity 3D-to-4D Scene Editing via Dynamic Propagation
State Key Laboratory of Complex and Critical Software Environment, Beihang University（北京航空航天大学复杂关键软件环境国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Zhejiang Industrial Big Data and Robot Intelligent System Key Laboratory, Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院浙江省工业大数据与机器人智能系统重点实验室）
DisCo3D: Distilling Multi-View Consistency for 3D Scene Editing
Institute for Artificial Intelligence, Beijing National Research Center for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University（人工智能研究院、北京信息科学与技术国家研究中心、计算机科学与技术系、清华大学）；College of Computer, National University of Defense Technology（计算机学院、国防科技大学）
We present a neural rendering framework for simultaneous view synthesis and appearance editing of a scene from multi-view images captured under known environment illumination. Existing approaches either achieve view synthesis alone or view synthesis along with relighting, without direct control over the scene's appearance. Our approach explicitly disentangles the appearance and learns a lighting representation that is independent of it. Specifically, we independently estimate the BRDF and use it to learn a lighting-only representation of the scene. Such disentanglement allows our approach to g…
UC San Diego（加州大学圣地亚哥分校）；Carnegie Mellon University（卡内基梅隆大学）；Adobe Research（Adobe研究院）
We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. This is an extremely challenging problem that requires modeling complex light transport, and disentangling HDR lighting from material and geometry with only a partial LDR observation of the scene. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions. We use physically-based indoor …
ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editing
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Pengcheng Laboratory, China（鹏城实验室）；Harbin Institute of Technology, China（哈尔滨工业大学）；Harbin Institute of Technology, Suzhou Research Institute, China（哈尔滨工业大学苏州市研究院）
3D-aware image synthesis has attracted increasing interest as it models the 3D nature of our real world. However, performing realistic object-level editing of the generated images in the multi-object scenario still remains a challenge. Recently, a 2D GAN termed BlobGAN has demonstrated great multi-object editing capabilities on real-world indoor scene datasets. In this work, we propose BlobGAN-3D, which is a 3D-aware improvement of the original 2D BlobGAN. We enable explicit camera pose control while maintaining the disentanglement for individual objects in the scene by extending the 2D blobs …
GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing
MT Lab, Meitu Inc.（美图公司MT实验室）；Department of Computer Science and Technology, BNRist, IDG/McGovern Institute for Brain Research, Tsinghua University（清华大学计算机科学与技术系）
ETH Zurich（苏黎世联邦理工学院）；CSEM SA（瑞士电子与微技术中心股份公司）；KU Leuven（鲁汶大学）
Recent advances in image generation gave rise to powerful tools for semantic image editing. However, existing approaches can either operate on a single image or require an abundance of additional information. They are not capable of handling the complete set of editing operations, that is addition, manipulation or removal of semantic concepts. To address these limitations, we propose SESAME, a novel generator-discriminator pair for Semantic Editing of Scenes by Adding, Manipulating or Erasing objects. In our setup, the user provides the semantic labels of the areas to be edited and the generat…
DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes
Huazhong University of Science and Technology（华中科技大学）；National Key Laboratory of Multispectral Information Intelligent Processing Technology（多光谱信息智能处理技术国家重点实验室）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing
Institute of Computing Technology, Chinese Academy of Sciences, China（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Hong Kong University of Science（香港科学大学）
InterCoG: Towards Spatially Precise Image Editing with Interleaved Chain-of-Grounding Reasoning
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；Zhengzhou Advanced Research Institute of Harbin Institute of Technology（哈尔滨工业大学郑州先进研究院）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术大学人工智能研究院）
TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；College of Computer Science, Nankai University（南开大学计算机学院）
University of Missouri（密苏里大学）；Microsoft Research（微软研究院）
From a single picture of a scene, people can typically grasp the spatial layout immediately and even make good guesses at materials properties and where light is coming from to illuminate the scene. For example, we can reliably tell which objects occlude others, what an object is made of and its rough shape, regions that are illuminated or in shadow, and so on. It is interesting how little is known about our ability to make these determinations; as such, we are still not able to robustly "teach" computers to make the same high-level observations as people. This document presents algorithms for…
