Unsupervised Region-Based Image Editing of Denoising Diffusion Models
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Pengcheng Laboratory（鹏城实验室）；University of Trento（特伦托大学）；Lanzhou Jiaotong University（兰州交通大学）
Zixiang Li, Yue Song, Renshuai Tao, Xiaohong Jia, Yao Zhao, Wei Wang
Although diffusion models have achieved remarkable success in the field of image generation, their latent space remains under-explored. Current methods for identifying semantics within latent space often rely on external supervision, such as textual information and segmentation masks. In this paper, we propose a method to identify semantic attributes in the latent space of pre-trained diffusion models without any further training. By projecting the Jacobian of the targeted semantic region into a low-dimensional subspace which is orthogonal to the non-masked regions, our approach facilitates precise semantic discovery and control over local masked areas, eliminating the need for annotations. We conducted extensive experiments across multiple datasets and various architectures of diffusion models, achieving state-of-the-art performance. In particular, for some specific face attributes, the performance of our proposed method even surpasses that of supervised approaches, demonstrating its superior ability in editing local image properties.
Pengcheng Laboratory（鹏城实验室）；Shenzhen University（深圳大学）；University of Nottingham（诺丁汉大学）
Facial attributes in StyleGAN generated images are entangled in the latent space which makes it very difficult to independently control a specific attribute without affecting the others. Supervised attribute editing requires annotated training data which is difficult to obtain and limits the editable attributes to those with labels. Therefore, unsupervised attribute editing in an disentangled latent space is key to performing neat and versatile semantic face editing. In this paper, we present a new technique termed Structure-Texture Independent Architecture with Weight Decomposition and Orthog…
DECADE: A Temporally-Consistent Unsupervised Diffusion Model for Enhanced Rb-82 Dynamic Cardiac PET Image Denoising
Yale University（耶鲁大学）；Yale School of Medicine（耶鲁医学院）；Siemens Medical Solutions USA, Inc.（西门子医疗解决方案美国公司）；Inselspital, Bern University Hospital（伯尔尼大学医院Inselspital）；University of Bern（伯尔尼大学）；Department of Biomedical Engineering（生物医学工程系）；Department of Radiology and Biomedical Imaging（放射科和生物医学成像系）；Department of Medicine (Cardiology)（医学系（心脏病学））
FDS: Frequency-Aware Denoising Score for Text-Guided Latent Diffusion Image Editing
School of Computer and Communication Sciences, EPFL（EPFL计算机与通信科学学院）；Department of Electrical and Photonics Engineering, DTU（丹麦技术大学电气与光子工程系）；Department of Electrical Engineering, Chalmers University of Technology（查尔姆斯理工大学电气工程系）
SPIN: Image Immunization Against Diffusion Editing via Single-Step Projection in Stochastic Neighborhoods
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Imperial College London（伦敦帝国理工学院）
University of Science and Technology of China（中国科学技术大学）；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Aarhus University（奥胡斯大学）
Seoul National University（首尔国立大学）；Yonsei University（延世大学）
Despite the success of diffusion models (DMs), we still lack a thorough understanding of their latent space. While image editing with GANs builds upon latent space, DMs rely on editing the conditions such as text prompts. We present an unsupervised method to discover interpretable editing directions for the latent variables $\mathbf{x}_t \in \mathcal{X}$ of DMs. Our method adopts Riemannian geometry between $\mathcal{X}$ and the intermediate feature maps $\mathcal{H}$ of the U-Nets to provide a deep understanding over the geometrical structure of $\mathcal{X}$. The discovered semantic latent d…
GAN Inversion for Image Editing via Unsupervised Domain Adaptation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Virginia（弗吉尼亚大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Nanjing University（南京大学）；Tsinghua University（清华大学）；SenseTime Research（商汤科技研究院）
Speech Driven Video Editing via an Audio-Conditioned Diffusion Model
University of Galway（高威大学）；University of Wrocław（弗罗茨瓦夫大学）；Wrocław University of Science and Technology（弗罗茨瓦夫理工大学）；Tooploox（Tooploox公司）；Trinity College Dublin（都柏林圣三一学院）
Criteo AI Lab（科睿拓人工智能实验室）；Ecole des Ponts（巴黎桥路学院）；Univ Gustave Eiffel（古斯塔夫·埃菲尔大学）；CNRS（法国国家科学研究中心）
Advances in computer vision are pushing the limits of im-age manipulation, with generative models sampling detailed images on various tasks. However, a specialized model is often developed and trained for each specific task, even though many image edition tasks share similarities. In denoising, inpainting, or image compositing, one always aims at generating a realistic image from a low-quality one. In this paper, we aim at making a step towards a unified approach for image editing. To do so, we propose EdiBERT, a bi-directional transformer trained in the discrete latent space built by a vector…
Multi-task SAR Image Processing via GAN-based Unsupervised Manipulation
School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Kunshan Innovation Institute of Xidian University（西安电子科技大学昆山创新研究院）；University of Montenegro（黑山大学）
As recent generative models can generate photo-realistic images, people seek to understand the mechanism behind the generation process. Interpretable generation process is beneficial to various image editing applications. In this work, we propose a framework to discover interpretable directions in the latent space given arbitrary pre-trained generative adversarial networks. We propose to learn the transformation from prior one-hot vectors representing different attributes to the latent space used by pre-trained models. Furthermore, we apply a centroid loss function to improve consistency and s…
Towards Consistent and Controllable Image Synthesis for Face Editing
Center for Machine Vision and Signal Analysis, Faculty of Information Technology and Electrical Engineering, University of Oulu（机器视觉与信号分析中心，信息科技与电气工程学院，奥卢大学）；Key Laboratory of Child Development and Learning Science of Ministry of Education, School of Biological Sciences and Medical Engineering, Southeast University（教育部儿童发展与学习科学重点实验室，生物科学与医学工程学院，东南大学）
Enabling Local Editing in Diffusion Models by Joint and Individual Component Analysis
National Technical University of Athens（雅典国家技术大学）；Archimedes AI, Athena RC（雅典娜研究中心阿基米德人工智能）；National and Kapodistrian University of Athens（雅典国立卡波季斯特里安大学）；The Cyprus Institute（塞浦路斯研究所）
Denoising Diffusion models are gaining increasing popularity in the field of generative modeling for several reasons, including the simple and stable training, the excellent generative quality, and the solid probabilistic foundation. In this article, we address the problem of {\em embedding} an image into the latent space of Denoising Diffusion Models, that is finding a suitable ``noisy'' image whose denoising results in the original image. We particularly focus on Denoising Diffusion Implicit Models due to the deterministic nature of their reverse diffusion process. As a side result of our in…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Recently, CycleGAN was shown to provide high-performance, ultra-fast denoising for low-dose X-ray computed tomography (CT) without the need for a paired training dataset. Although this was possible thanks to cycle consistency, CycleGAN requires two generators and two discriminators to enforce cycle consistency, demanding significant GPU resources and technical skills for training. A recent proposal of tunable CycleGAN with Adaptive Instance Normalization (AdaIN) alleviates the problem in part by using a single generator. However, two discriminators and an additional AdaIN code generator are st…
Reversible Inversion for Training-Free Exemplar-guided Image Editing
school of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；School of Computing and Artificial Intelligence, Southwest Jiaotong University, China（西南交通大学计算机科学与人工智能学院）；School of Computer Science and Technology, Tongji University, China（同济大学计算机科学与技术学院）；Department of Computer Science, University of Warwick, United Kingdom（英国沃里克大学计算机科学系）
Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing
Media Integration and Communication Center (MICC), University of Florence（佛罗伦萨大学媒体集成与传播中心（MICC））；University of Florence（佛罗伦萨大学）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；University of Pisa（比萨大学）
Tuning-Free Latent Diffusion Models for Ultrahigh-Resolution Image Editing
College of Computer Science and Artificial Intelligence, Wenzhou University（温州大学计算机科学与人工智能学院）；Department of Computer Science, Memorial University of Newfoundland（纽芬兰纪念大学计算机科学系）；AI Analytics Team, Nasdaq（纳斯达克人工智能分析团队）；State Key Laboratory of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）
