Diffusion models gain increasing popularity for their generative capabilities. Recently, there have been surging needs to generate customized images by inverting diffusion models from exemplar images, and existing inversion methods mainly focus on capturing object appearances (i.e., the "look"). However, how to invert object relations, another important pillar in the visual world, remains unexplored. In this work, we propose the Relation Inversion task, which aims to learn a specific relation (represented as "relation prompt") from exemplar images. Specifically, we learn a relation prompt with a frozen pre-trained text-to-image diffusion model. The learned relation prompt can then be applied to generate relation-specific images with new objects, backgrounds, and styles.
To tackle the Relation Inversion task, we propose the ReVersion Framework. Specifically, we propose a novel "relation-steering contrastive learning" scheme to steer the relation prompt towards relation-dense regions, and disentangle it away from object appearances. We further devise "relation-focal importance sampling" to emphasize high-level interactions over low-level appearances (e.g., texture, color). To comprehensively evaluate this new task, we contribute the ReVersion Benchmark, which provides various exemplar images with diverse relations. Extensive experiments validate the superiority of our approach over existing methods across a wide range of visual relations. Our proposed task and method could be good inspirations for future research in various domains like generative inversion, few-shot learning, and visual relation detection.
TINA+: Probing Residual Visual Knowledge in Unlearned Diffusion Models via Diffusion-Consistent Text-Free Inversion
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；City University of Hong Kong（香港城市大学）；Shenzhen Loop Area Institute（深圳河套学院）；National University of Singapore（新加坡国立大学）；Pengcheng Laboratory（鹏城实验室）
OSI: One-step Inversion Excels in Extracting Diffusion Watermarks
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, China（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences (CAS), China（中国科学院大学）；School of Information Science and Technology, ShanghaiTech University, China（信息科学与技术学院，上海科技大学）
DCI: Dual-Conditional Inversion for Boosting Diffusion-Based Image Editing
Institute of Information Science, Beijing Jiaotong University（信息科学学院，北京交通大学）；Visual Intelligence +X International Cooperation Joint Laboratory of MOE（视觉智能+X国际合作联合实验室）
Image Inversion: A Survey from GANs to Diffusion and Beyond
College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；YouTu Lab, Tencent（腾讯优图实验室）；College of Computer and Information Science, Southwest University（西南大学计算机与信息科学学院）；Department of Computer Science & Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
Training-free image inversion for one-step diffusion models
CVC, University of Alabama in Birmingham（CVC，阿拉巴马大学伯明翰分校）；Machine Intelligence Institute, Masdar Institute of Science and Technology（机器智能研究所，马斯达尔科技 institute）；Jilin University（吉林大学）；City University of Hong Kong, Department of Geography（香港城市大学地理系）
Compositional Inversion for Stable Diffusion Models
the Hong Kong Polytechnic University（香港理工大学）；Center for Artificial Intelligence and Robotics, HKISI, CAS（中国科学院香港创新研究院人工智能与机器人创新中心）；Sichuan University（四川大学）；UCAS（中国科学院大学）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（中国科学院自动化研究所多模态人工智能系统国家重点实验室）
Task-Oriented Diffusion Inversion for High-Fidelity Text-based Editing
The University of Hong Kong（香港大学）；Shanghai AI Lab（上海人工智能实验室）；Ocean University of China（中国海洋大学）；Singapore Management University（新加坡管理大学）；South China University of Technology（华南理工大学）
Latent Bias Alignment for High-Fidelity Diffusion Inversion in Real-World Image Reconstruction and Manipulation
Department of Electrical and Electronic Engineering, Southern University of Science and Technology（南方科技大学电子与电气工程系）；Undergraduate School of Artificial Intelligence, Shenzhen Polytechnic University（深圳理工大学人工智能本科生学院）；Huawei Technologies Co., Ltd.（华为技术有限公司）；Pengcheng Lab（鹏城实验室）
EDITOR: Effective and Interpretable Prompt Inversion for Text-to-Image Diffusion Models
University of Massachusetts, Amherst（马萨诸塞大学阿姆赫斯特分校）；Georgia Institute of Technology（佐治亚理工学院）；Rutgers University（罗格斯大学）；University of Utah（犹他大学）；Dolby Laboratories（杜比实验室）
TINA: Text-Free Inversion Attack for Unlearned Text-to-Image Diffusion Models
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；City University of Hong Kong（香港城市大学）；Shenzhen Loop Area Institute（深圳河套学院）；Peng Cheng Laboratory（鹏城实验室）；Shandong University（山东大学）
Model Inversion Attacks Through Target-Specific Conditional Diffusion Models
School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）
There and Back Again: On the relation between Noise and Image Inversions in Diffusion Models
Warsaw University of Technology（华沙技术大学）；IDEAS Research Institute（IDEAS研究机构）；University of Warsaw（华沙大学）；Institute of Mathematics, Polish Academy of Sciences（波兰科学院数学研究所）；IDEAS NCBR
UniEdit-Flow: Unleashing Inversion and Editing in the Era of Flow Models
The University of British Columbia（不列颠哥伦比亚大学）；Tsinghua University（清华大学）；Snap Inc.（Snap公司）；Vector Institute（向量研究所）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）
Conditioning Matters: Stabilizing Inversion and Attention in Diffusion Image Editing
State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全全国重点实验室）；HangZhou High-Tech Zong (Binjiang) Institute of Blockchain and Data Security（杭州高新技术产业开发区（滨江）区块链与数据安全研究院）；College of Computer Science, Zhejiang University（浙江大学计算机科学与技术学院）；University at Buffalo, State University of New York（纽约州立大学布法罗分校）
DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models
Rutgers University（新泽西罗格斯大学）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；Red Hat AI Innovation（红帽AI创新）；Google DeepMind（谷歌DeepMind）；NYU（纽约大学）；Walmart Global Tech（沃尔玛全球技术）；NEC Labs America（NEC美国实验室）；Massachusetts Institute of Technology（麻省理工学院）；ANU（澳大利亚国立大学）；UT Arlington（德克萨斯大学阿灵顿分校）
