Existing text-to-image editing methods tend to excel either in rigid or non-rigid editing but encounter challenges when combining both, resulting in misaligned outputs with the provided text prompts. In addition, integrating reference images for control remains challenging. To address these issues, we present a versatile image editing framework capable of executing both rigid and non-rigid edits, guided by either textual prompts or reference images. We leverage a dual-path injection scheme to handle diverse editing scenarios and introduce an integrated self-attention mechanism for fusion of appearance and structural information. To mitigate potential visual artifacts, we further employ latent fusion techniques to adjust intermediate latents. Compared to previous work, our approach represents a significant advance in achieving precise and versatile image editing. Comprehensive experiments validate the efficacy of our method, showcasing competitive or superior results in text-based editing and appearance transfer tasks, encompassing both rigid and non-rigid settings.
Image generation has recently seen tremendous advances, with diffusion models allowing to synthesize convincing images for a large variety of text prompts. In this article, we propose DiffEdit, a method to take advantage of text-conditioned diffusion models for the task of semantic image editing, where the goal is to edit an image based on a text query. Semantic image editing is an extension of image generation, with the additional constraint that the generated image should be as similar as possible to a given input image. Current editing methods based on diffusion models usually require to pr…
ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motions
ByteDance Seed（字节跳动种子）；University of Southern California（南加州大学）；University of Tokyo（东京大学）；University of California Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；University of California Los Angeles（加州大学洛杉矶分校）
EleutherAI；Booz Allen Hamilton（博思艾伦汉密尔顿）；AIDock；Georgia Institute of Technology（佐治亚理工学院）
Generating and editing images from open domain text prompts is a challenging task that heretofore has required expensive and specially trained models. We demonstrate a novel methodology for both tasks which is capable of producing images of high visual quality from text prompts of significant semantic complexity without any training by using a multimodal encoder to guide image generations. We demonstrate on a variety of tasks how using CLIP [37] to guide VQGAN [11] produces higher visual quality outputs than prior, less flexible approaches like DALL-E [38], GLIDE [33] and Open-Edit [24], despi…
Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing
HSE University（高等经济大学）；AIRI（人工智能研究中心）；Constructor University（康斯特大学）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）；UNSW Sydney（新南威尔士大学悉尼分校）
When Test-Time Guidance Is Enough: Fast Image and Video Editing with Diffusion Guidance
CMAP, Ecole Polytechnique（巴黎综合理工学院CMAP实验室）；Institute of Foundation Models（基础模型研究所）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；EPITA（法国高等信息技术学院）；Sorbonne University（索邦大学）；Lagrange Mathematics and Computing Research Center（拉格朗日数学与计算研究中心）；EPITA Research Lab（EPITA研究实验室）；KTH Royal Institute of Technology（瑞典皇家理工学院）
TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；College of Computer Science, Nankai University（南开大学计算机学院）
Tuning-free Instruction-based Video Editing Via Structural Noise Initialization and Guidance
JIUTIAN Research, China Mobile（中国移动极天研究院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；State Key Laboratory of Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）
Multimodal Image Synthesis and Editing: The Generative AI Era
Max Planck Institute for Informatics（马克斯·普朗克信息研究所）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）；Nanyang Technological University（南洋理工大学）；Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Object-Aware Background-Controlled Editing via Weighted Velocity Guidance
Carnegie Mellon University（卡内基梅隆大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
The Chinese University of Hong Kong（香港中文大学）；Adobe（奥多比）；SmartMore（思谋）
This paper presents Video-P2P, a novel framework for real-world video editing with cross-attention control. While attention control has proven effective for image editing with pre-trained image generation models, there are currently no large-scale video generation models publicly available. Video-P2P addresses this limitation by adapting an image generation diffusion model to complete various video editing tasks. Specifically, we propose to first tune a Text-to-Set (T2S) model to complete an approximate inversion and then optimize a shared unconditional embedding to achieve accurate video inve…
