ConsistDreamer：面向高保真场景编辑的3D一致2D扩散
ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editing
AI总结：
本文提出 ConsistDreamer 框架，通过环绕视角输入、3D一致结构化噪声与自监督一致性训练，使2D扩散模型实现高保真3D一致的指令引导场景编辑。
摘要：
本文提出 ConsistDreamer——一种新颖框架，它赋予2D扩散模型3D感知与3D一致性，从而实现高保真的指令引导场景编辑。为克服2D扩散模型缺失3D一致性这一根本局限，我们的核心洞见是引入三种协同策略，使2D扩散模型的输入增强为具备3D感知，并在训练过程中显式施加3D一致性。具体而言，我们将环绕视角设计为2D扩散模型上下文丰富的输入，并生成3D一致的结构化噪声，而非与图像无关的噪声。此外，我们在逐场景编辑流程中引入自监督的一致性强制训练。大量评估表明，ConsistDreamer 在各类场景与编辑指令下均取得指令引导场景编辑的最先进性能，尤其在来自 ScanNet++ 的复杂大规模室内场景中，清晰度与细粒度纹理得到显著提升。值得注意的是，ConsistDreamer 是首个能够成功编辑复杂图案（例如格子/棋盘格图案）的工作。项目页面位于 immortalco.github.io/ConsistDreamer。
英文摘要：
This paper proposes ConsistDreamer - a novel framework that lifts 2D diffusion models with 3D awareness and 3D consistency, thus enabling high-fidelity instruction-guided scene editing. To overcome the fundamental limitation of missing 3D consistency in 2D diffusion models, our key insight is to introduce three synergetic strategies that augment the input of the 2D diffusion model to become 3D-aware and to explicitly enforce 3D consistency during the training process. Specifically, we design surrounding views as context-rich input for the 2D diffusion model, and generate 3D-consistent, structured noise instead of image-independent noise. Moreover, we introduce self-supervised consistency-enforcing training within the per-scene editing procedure. Extensive evaluation shows that our ConsistDreamer achieves state-of-the-art performance for instruction-guided scene editing across various scenes and editing instructions, particularly in complicated large-scale indoor scenes from ScanNet++, with significantly improved sharpness and fine-grained textures. Notably, ConsistDreamer stands as the first work capable of successfully editing complex (e.g., plaid/checkered) patterns. Our project page is at immortalco.github.io/ConsistDreamer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
