通过比较原始提示词和目标提示词，我们可以获得大量的编辑对，每个编辑对包含一个对象及其对应的编辑目标。为了在保持对输入图像保真度的同时实现可编辑性，现有的编辑方法通常采用固定数量的反演步骤，将整个输入图像投影到其噪声更大的潜在表示中，随后由目标提示词引导的去噪过程。然而，我们发现，由于编辑难度各异，不同编辑对实现理想编辑结果所需的最优反演步骤数差异显著。因此，当前依赖固定反演步骤数的文献，在生成质量上产生次优结果，尤其是在处理自然图像中的多个编辑对时。为此，我们提出了一种新的图像编辑范式，称为对象感知反演与重组（Object-aware Inversion and Reassembly, OIR），以实现对象级别的细粒度编辑。具体而言，我们设计了一种新的搜索度量，通过联合考虑目标的可编辑性和非编辑区域的保真度，为每个编辑对确定最优反演步骤数。在编辑图像时，我们使用该搜索度量为每个编辑对找到最优反演步骤。然后，我们分别编辑这些编辑对以避免概念不匹配。随后，我们提出一个额外的重组步骤，将各自的编辑结果与非编辑区域无缝整合，以获得最终的编辑图像。为系统评估我们方法的有效性，我们收集了两个数据集，分别称为OIRBench，用于基准测试单对象和多对象编辑。实验表明，我们的方法在编辑对象形状、颜色、材质、类别等方面表现出优越性能，尤其是在多对象编辑场景中。
英文摘要：
By comparing the original and target prompts, we can obtain numerous editing pairs, each comprising an object and its corresponding editing target. To allow editability while maintaining fidelity to the input image, existing editing methods typically involve a fixed number of inversion steps that project the whole input image to its noisier latent representation, followed by a denoising process guided by the target prompt. However, we find that the optimal number of inversion steps for achieving ideal editing results varies significantly among different editing pairs, owing to varying editing difficulties. Therefore, the current literature, which relies on a fixed number of inversion steps, produces sub-optimal generation quality, especially when handling multiple editing pairs in a natural image. To this end, we propose a new image editing paradigm, dubbed Object-aware Inversion and Reassembly (OIR), to enable object-level fine-grained editing. Specifically, we design a new search metric, which determines the optimal inversion steps for each editing pair, by jointly considering the editability of the target and the fidelity of the non-editing region. We use our search metric to find the optimal inversion step for each editing pair when editing an image. We then edit these editing pairs separately to avoid concept mismatch. Subsequently, we propose an additional reassembly step to seamlessly integrate the respective editing results and the non-editing region to obtain the final edited image. To systematically evaluate the effectiveness of our method, we collect two datasets called OIRBench for benchmarking single- and multi-object editing, respectively. Experiments demonstrate that our method achieves superior performance in editing object shapes, colors, materials, categories, etc., especially in multi-object editing scenarios.
DCI: Dual-Conditional Inversion for Boosting Diffusion-Based Image Editing
Institute of Information Science, Beijing Jiaotong University（信息科学学院，北京交通大学）；Visual Intelligence +X International Cooperation Joint Laboratory of MOE（视觉智能+X国际合作联合实验室）
Google Research（谷歌研究院）；The Blavatnik School of Computer Science, Tel Aviv University（特拉维夫大学布拉瓦尼克计算机科学学院）
Recent text-guided diffusion models provide powerful image generation capabilities. Currently, a massive effort is given to enable the modification of these images using text only as means to offer intuitive and versatile editing. To edit a real image using these state-of-the-art tools, one must first invert the image with a meaningful text prompt into the pretrained model's domain. In this paper, we introduce an accurate inversion technique and thus facilitate an intuitive text-based modification of the image. Our proposed inversion consists of two novel key components: (i) Pivotal inversion …
The Chinese University of Hong Kong（香港中文大学）；Xiaomi AI Lab（小米人工智能实验室）
Recent work has shown that a variety of semantics emerge in the latent space of Generative Adversarial Networks (GANs) when being trained to synthesize images. However, it is difficult to use these learned semantics for real image editing. A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code. However, existing inversion methods typically focus on reconstructing the target image by pixel values yet fail to land the inverted code in the semantic domain of the original latent space. As a result, the reconstructed image cannot well support sema…
University of Chinese Academy of Sciences（中国科学院大学）；Center for Research on Intelligent Perception and Computing, NLPR, CASIA（中国科学院自动化研究所模式识别国家重点实验室智能感知与计算研究中心）
GAN inversion has been exploited in many face manipulation tasks, but 2D GANs often fail to generate multi-view 3D consistent images. The encoders designed for 2D GANs are not able to provide sufficient 3D information for the inversion and editing. Therefore, 3D-aware GAN inversion is proposed to increase the 3D editing capability of GANs. However, the 3D-aware GAN inversion remains under-explored. To tackle this problem, we propose a 3D-aware (3Da) encoder for GAN inversion and face editing based on the powerful StyleNeRF model. Our proposed 3Da encoder combines a parametric 3D face model wit…
UniEdit-Flow: Unleashing Inversion and Editing in the Era of Flow Models
The University of British Columbia（不列颠哥伦比亚大学）；Tsinghua University（清华大学）；Snap Inc.（Snap公司）；Vector Institute（向量研究所）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）
Task-Oriented Diffusion Inversion for High-Fidelity Text-based Editing
The University of Hong Kong（香港大学）；Shanghai AI Lab（上海人工智能实验室）；Ocean University of China（中国海洋大学）；Singapore Management University（新加坡管理大学）；South China University of Technology（华南理工大学）
DirectEdit: Step-Level Accurate Inversion for Flow-Based Image Editing
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, Wuhan, China（多媒体软件国家工程研究中心，计算机科学学院，武汉大学，中国武汉）
We present a novel image inversion framework and a training pipeline to achieve high-fidelity image inversion with high-quality attribute editing. Inverting real images into StyleGAN's latent space is an extensively studied problem, yet the trade-off between the image reconstruction fidelity and image editing quality remains an open challenge. The low-rate latent spaces are limited in their expressiveness power for high-fidelity reconstruction. On the other hand, high-rate latent spaces result in degradation in editing quality. In this work, to achieve high-fidelity inversion, we learn residua…
GAN Inversion for Image Editing via Unsupervised Domain Adaptation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Virginia（弗吉尼亚大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Harvard AI and Robotics Lab, Harvard University（哈佛人工智能与机器人实验室，哈佛大学）；École des Ponts, Institut Polytechnique de Paris（巴黎高等工业学院）；New York University Abu Dhabi（纽约大学阿布扎克分校）；Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University（哈佛大学自然与人工智能研究学院）
Rectified flow models have become a de facto standard in image generation due to their stable sampling trajectories and high-fidelity outputs. Despite their strong generative capabilities, they face critical limitations in image editing tasks: inaccurate inversion processes for mapping real images back into the latent space, and gradient entanglement issues during editing often result in outputs that do not faithfully reflect the target prompt. Recent efforts have attempted to directly map source and target distributions via ODE-based approaches without inversion; however,these methods still y…
For successful semantic editing of real images, it is critical for a GAN inversion method to find an in-domain latent code that aligns with the domain of a pre-trained GAN model. Unfortunately, such in-domain latent codes can be found only for in-range images that align with the training images of a GAN model. In this paper, we propose BDInvert, a novel GAN inversion approach to semantic editing of out-of-range images that are geometrically unaligned with the training images of a GAN model. To find a latent code that is semantically editable, BDInvert inverts an input out-of-range image into a…
ProEdit: Inversion-based Editing From Prompts Done Right
Sun Yat-sen University（中山大学）；CUHK MMLab（香港中文大学MMLab）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；The University of Hong Kong（香港大学）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China（中国教育部机器智能与高级计算重点实验室）
South China University of Technology（华南理工大学）；Ocean University of China（中国海洋大学）；The Hong Kong Polytechnic University（香港理工大学）
Despite the demonstrated editing capacity in the latent space of a pretrained GAN model, inverting real-world images is stuck in a dilemma that the reconstruction cannot be faithful to the original input. The main reason for this is that the distributions between training and real-world data are misaligned, and because of that, it is unstable of GAN inversion for real image editing. In this paper, we propose a novel GAN prior based editing framework to tackle the out-of-domain inversion problem with a composition-decomposition paradigm. In particular, during the phase of composition, we introd…
LocInv: Localization-aware Inversion for Text-Guided Image Editing
University of Chinese Academy of Sciences（中国科学院大学）；Computer Vision Center（计算机视觉中心）；College of Computer Science（计算机学院）；Nankai University（南开大学）；Universitat Autonoma de Barcelona（巴塞罗那自治大学）
Shanghai Key Laboratory of Multidimensional Information Processing（上海多维度信息处理重点实验室）；Huazhong University of Science and Technology（华中科技大学）
Existing GAN inversion methods fail to provide latent codes for reliable reconstruction and flexible editing simultaneously. This paper presents a transformer-based image inversion and editing model for pretrained StyleGAN which is not only with less distortions, but also of high quality and flexibility for editing. The proposed model employs a CNN encoder to provide multi-scale image features as keys and values. Meanwhile it regards the style code to be determined for different layers of the generator as queries. It first initializes query tokens as learnable parameters and maps them into W+ …
The inversion of real images into StyleGAN's latent space is a well-studied problem. Nevertheless, applying existing approaches to real-world scenarios remains an open challenge, due to an inherent trade-off between reconstruction and editability: latent space regions which can accurately represent real images typically suffer from degraded semantic control. Recent work proposes to mitigate this trade-off by fine-tuning the generator to add the target image to well-behaved, editable regions of the latent space. While promising, this fine-tuning scheme is impractical for prevalent use as it req…
