arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.14792cs.CVcs.GRcs.LG
收藏

面向多视角图像编辑的注意力特征整合

Consolidating Attention Features for Multi-view Image Editing

Tel Aviv University(特拉维夫大学);NVIDIA(英伟达公司);Carnegie Mellon University(卡内基梅隆大学)

Or Patashnik, Rinon Gal, Daniel Cohen-Or, Jun-Yan Zhu, Fernando De la Torre

AI总结:

本文提出QNeRF方法,通过在编辑图像内部查询特征上训练神经辐射场并渲染三维一致查询,再将其注入自注意力层,从而提升多视角图像编辑的几何一致性和场景保真度。

摘要:

大规模文本到图像模型支持广泛的图像编辑技术,可使用文本提示甚至空间控制。然而,将这些编辑方法应用于描绘同一场景的多视角图像时,会导致三维不一致的结果。在本工作中,我们聚焦于基于空间控制的几何操作,并提出一种方法以整合跨多个视角的编辑过程。我们基于两点洞察:(1)在整个生成过程中保持特征一致有助于实现多视角编辑的一致性,以及(2)自注意力层中的查询(queries)对图像结构有显著影响。因此,我们提出通过强制查询的一致性来提升编辑后图像的几何一致性。为此,我们引入QNeRF,这是一个在编辑后图像的内部查询特征上训练的神经辐射场。训练完成后,QNeRF可以渲染出三维一致的查询,这些查询随后在生成过程中被软性注入回自注意力层,从而大幅提升多视角一致性。我们通过一种渐进式、迭代的方法来优化该过程,从而在扩散时间步上更好地整合查询。我们将本方法与一系列现有技术进行比较,并证明其能够实现更好的多视角一致性和对输入场景更高的保真度。这些优势使我们能够训练出视觉伪影更少、且与目标几何结构更对齐的NeRF。

英文摘要:

Large-scale text-to-image models enable a wide range of image editing techniques, using text prompts or even spatial controls. However, applying these editing methods to multi-view images depicting a single scene leads to 3D-inconsistent results. In this work, we focus on spatial control-based geometric manipulations and introduce a method to consolidate the editing process across various views. We build on two insights: (1) maintaining consistent features throughout the generative process helps attain consistency in multi-view editing, and (2) the queries in self-attention layers significantly influence the image structure. Hence, we propose to improve the geometric consistency of the edited images by enforcing the consistency of the queries. To do so, we introduce QNeRF, a neural radiance field trained on the internal query features of the edited images. Once trained, QNeRF can render 3D-consistent queries, which are then softly injected back into the self-attention layers during generation, greatly improving multi-view consistency. We refine the process through a progressive, iterative method that better consolidates queries across the diffusion timesteps. We compare our method to a range of existing techniques and demonstrate that it can achieve better multi-view consistency and higher fidelity to the input scene. These advantages allow us to train NeRFs with fewer visual artifacts, that are better aligned with the target geometry.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑