Edit-DiffNeRF：使用2D扩散模型编辑3D神经辐射场
Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model
AI总结：
针对扩散模型与NeRF简单耦合导致的跨视图不一致问题，提出Edit-DiffNeRF框架，通过delta模块编辑冻结扩散模型的潜在语义空间，结合多视图语义一致性损失，实现3D场景编辑与文本指令对齐度提升25%。
摘要：
近期研究表明，预训练扩散模型与神经辐射场（NeRF）的结合已成为文本到3D生成的一种颇具前景的方法。若简单将NeRF与扩散模型耦合，会导致跨视图不一致以及风格化视图合成质量下降的问题。为应对这一挑战，我们提出Edit-DiffNeRF框架，该框架由一个冻结的扩散模型、一个用于编辑扩散模型潜在语义空间的新增delta模块，以及一个NeRF组成。我们的方法无需针对每个场景训练整个扩散模型，而是聚焦于通过delta模块编辑冻结的预训练扩散模型中的潜在语义空间。这一对标准扩散框架的根本性改动，使我们能够对渲染视图进行细粒度修改，并通过NeRF训练将这些编辑指令有效整合到3D场景中，最终生成与输入文本指令精准对齐的编辑后3D场景。此外，为确保不同视角间的语义一致性，我们提出了一种新颖的多视图语义一致性损失：该损失从输入视图中提取潜在语义嵌入作为先验，并目标在不同视图中重建该嵌入。实验表明，我们提出的方法可有效编辑真实世界3D场景，与现有工作相比，所执行的3D编辑与文本指令的对齐度提升了25%。
英文摘要：
Recent research has demonstrated that the combination of pretrained diffusion models with neural radiance fields (NeRFs) has emerged as a promising approach for text-to-3D generation. Simply coupling NeRF with diffusion models will result in cross-view inconsistency and degradation of stylized view syntheses. To address this challenge, we propose the Edit-DiffNeRF framework, which is composed of a frozen diffusion model, a proposed delta module to edit the latent semantic space of the diffusion model, and a NeRF. Instead of training the entire diffusion for each scene, our method focuses on editing the latent semantic space in frozen pretrained diffusion models by the delta module. This fundamental change to the standard diffusion framework enables us to make fine-grained modifications to the rendered views and effectively consolidate these instructions in a 3D scene via NeRF training. As a result, we are able to produce an edited 3D scene that faithfully aligns to input text instructions. Furthermore, to ensure semantic consistency across different viewpoints, we propose a novel multi-view semantic consistency loss that extracts a latent semantic embedding from the input view as a prior, and aim to reconstruct it in different views. Our proposed method has been shown to effectively edit real-world 3D scenes, resulting in 25% improvement in the alignment of the performed 3D edits with text instructions compared to prior work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
