Recently, text-to-image (T2I) editing has been greatly pushed forward by applying diffusion models. Despite the visual promise of the generated images, inconsistencies with the expected textual prompt remain prevalent. This paper aims to systematically improve the text-guided image editing techniques based on diffusion models, by addressing their limitations. Notably, the common idea in diffusion-based editing firstly reconstructs the source image via inversion techniques e.g., DDIM Inversion. Then following a fusion process that carefully integrates the source intermediate (hidden) states (obtained by inversion) with the ones of the target image. Unfortunately, such a standard pipeline fails in many cases due to the interference of texture retention and the new characters creation in some regions. To mitigate this, we incorporate human annotation as an external knowledge to confine editing within a ``Mask-informed'' region. Then we carefully Fuse the edited image with the source image and a constructed intermediate image within the model's Self-Attention module. Extensive empirical results demonstrate the proposed ``MaSaFusion'' significantly improves the existing T2I editing techniques.
MADiff: Text-Guided Fashion Image Editing with Mask Prediction and Attention-Enhanced Diffusion
Southeast University（东南大学）；Advanced Ocean Institute of Southeast University（东南大学先进海洋研究院）；Northwestern Polytechnical University（西北工业大学）；Binjiang Institute of Artificial Intelligence, ZJUT（浙江工业大学滨江人工智能研究院）；Alibaba Group（阿里巴巴集团）
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）
Portrait editing is a popular subject in photo manipulation. The Generative Adversarial Network (GAN) advances the generating of realistic faces and allows more face editing. In this paper, we argue about three issues in existing techniques: diversity, quality, and controllability for portrait synthesis and editing. To address these issues, we propose a novel end-to-end learning framework that leverages conditional GANs guided by provided face masks for generating faces. The framework learns feature embeddings for every face component (e.g., mouth, hair, eye), separately, contributing to bette…
Image generation has recently seen tremendous advances, with diffusion models allowing to synthesize convincing images for a large variety of text prompts. In this article, we propose DiffEdit, a method to take advantage of text-conditioned diffusion models for the task of semantic image editing, where the goal is to edit an image based on a text query. Semantic image editing is an extension of image generation, with the additional constraint that the generated image should be as similar as possible to a given input image. Current editing methods based on diffusion models usually require to pr…
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
The success of Deep Generative Models at high-resolution image generation has led to their extensive utilization for style editing of real images. Most existing methods work on the principle of inverting real images onto their latent space, followed by determining controllable directions. Both inversion of real images and determination of controllable latent directions are computationally expensive operations. Moreover, the determination of controllable latent directions requires additional human supervision. This work aims to explore the efficacy of mask-guided feature modulation in the laten…
SAM-Flow: Source-Anchored Masked Flow for Training-Free Image Editing
Tianjin Key Laboratory of Imaging and Sensing Microelectronic Technology, School of Microelectronics, Tianjin University（天津影像与传感微电子技术重点实验室，微电子学院，天津大学）；School of Cyber Security, Tianjin University（网络安全学院，天津大学）
MDE-Edit: Masked Dual-Editing for Multi-Object Image Editing via Diffusion Models
School of Computer Science and Information Engineering（计算机科学与信息工程学院）；Hefei University of Technology（合肥工业大学）；School of computer and artificial intelligence（计算机与人工智能学院）；Liaoning Normal University（辽宁师范大学）
Face editing represents a popular research topic within the computer vision and image processing communities. While significant progress has been made recently in this area, existing solutions: (i) are still largely focused on low-resolution images, (ii) often generate editing results with visual artefacts, or (iii) lack fine-grained control and alter multiple (entangled) attributes at once, when trying to generate the desired facial semantics. In this paper, we aim to address these issues though a novel attribute editing approach called MaskFaceGAN that focuses on local attribute editing. The…
AnyDesign: Versatile Area Fashion Editing via Mask-Free Diffusion
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Wuhan AI Research（武汉人工智能研究所）；Wuhan University（武汉大学）
AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner
School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；State Key Lab of Multimedia Info. Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；Nat’l Eng. Research Ctr. of Visual Tech., School of Computer Science, Peking University（北京大学计算机学院视觉技术工程研究中心）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
School of Computer and Information Technology, Beijing Jiaotong University（北京交通大学计算机与信息技术学院）
Image synthesis has witnessed substantial progress due to the increasing power of generative model. This paper we propose a novel generative approach for exemplar based facial editing in the form of the region inpainting. Our method first masks the facial editing region to eliminates the pixel constraints of the original image, then exemplar based facial editing can be achieved by learning the corresponding information from the reference image to complete the masked region. In additional, we impose the attribute labels constraint to model disentangled encodings in order to avoid undesired info…
ASTRA: Let Arbitrary Subjects Transform in Video Editing
NExT++ Research Centre, National University of Singapore（新加坡国立大学NExT++研究中心）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；College of Information Science and Technology and Artificial Intelligence, Nanjing Forestry University（南京林业大学信息科学与技术学院及人工智能学院）
Imperial College London（帝国理工学院）；University College London（伦敦大学学院）；University of Sheffield（谢菲尔德大学）；Korea Advanced Institute of Science and Technology（韩国高等科学技术研究院）
Semantically guided conditional Generative Adversarial Networks (cGANs) have become a popular approach for face editing in recent years. However, most existing methods introduce semantic masks as direct conditional inputs to the generator and often require the target masks to perform the corresponding translation in the RGB space. We propose SeCGAN, a novel label-guided cGAN for editing face images utilising semantic information without the need to specify target semantic masks. During training, SeCGAN has two branches of generators and discriminators operating in parallel, with one trained to…
University at Albany, State University of New York, USA（纽约州立大学奥尔巴尼分校）；Microsoft Corporation, Redmond, USA（微软公司雷德蒙德分部）；Samsung Research America AI Center, USA（三星美国研究公司人工智能中心）；University at Buffalo, State University of New York, USA（纽约州立大学布法罗分校）
We present Mask-guided Generative Adversarial Network (MagGAN) for high-resolution face attribute editing, in which semantic facial masks from a pre-trained face parser are used to guide the fine-grained image editing process. With the introduction of a mask-guided reconstruction loss, MagGAN learns to only edit the facial parts that are relevant to the desired attribute changes, while preserving the attribute-irrelevant regions (e.g., hat, scarf for modification `To Bald'). Further, a novel mask-guided conditioning strategy is introduced to incorporate the influence region of each attribute c…
Edit Where You Mean: Region-Aware Adapter Injection for Mask-Free Local Image Editing
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Beijing University of Aeronautics and Astronautics（北京航空航天大学）；Tsinghua University（清华大学）；Peking University（北京大学）；Xiaohongshu Inc.（小红书公司）
DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models
Rutgers University（新泽西罗格斯大学）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；Red Hat AI Innovation（红帽AI创新）；Google DeepMind（谷歌DeepMind）；NYU（纽约大学）；Walmart Global Tech（沃尔玛全球技术）；NEC Labs America（NEC美国实验室）；Massachusetts Institute of Technology（麻省理工学院）；ANU（澳大利亚国立大学）；UT Arlington（德克萨斯大学阿灵顿分校）
Tsinghua University（清华大学）；Tencent AI Lab（腾讯人工智能实验室）；ByteDance Inc.（字节跳动公司）
Existing 3D-aware facial generation methods face a dilemma in quality versus editability: they either generate editable results in low resolution or high-quality ones with no editing flexibility. In this work, we propose a new approach that brings the best of both worlds together. Our system consists of three major components: (1) a 3D-semantics-aware generative model that produces view-consistent, disentangled face images and semantic masks; (2) a hybrid GAN inversion approach that initialize the latent codes from the semantic and texture encoder, and further optimized them for faithful recon…
