Latent diffusion models such as Stable Diffusion achieve state-of-the-art results on text-to-image generation tasks. However, the extent to which these models have a semantic understanding of the images they generate is not well understood. In this work, we investigate whether the internal representations used by these models during text-to-image generation contain semantic information that is meaningful to humans. To do so, we perform probing on Stable Diffusion with simple regression layers that predict semantic attributes for objects and evaluate these predictions against human annotations. Surprisingly, we find that this success can actually be attributed to the text encoding occurring in CLIP rather than the reverse diffusion process. We demonstrate that groups of specific semantic attributes have markedly different decoding accuracy than the average, and are thus represented to different degrees. Finally, we show that attributes become more difficult to disambiguate from one another during the inverse diffusion process, further demonstrating the strongest semantic representation of object attributes in CLIP. We conclude that the separately trained CLIP vision-language model is what determines the human-like semantic representation, and that the diffusion process instead takes the role of a visual decoder.
Semantic Relation-Enhanced CLIP Adapter for Domain Adaptive Zero-Shot Learning
School of Computer Science and Technology, East China Normal University, China（东华大学计算机科学与技术学院）；College of Computer Science and Technology, Ocean University of China, China（中国海洋大学计算机科学与技术学院）
FB-CLIP: Fine-Grained Zero-Shot Anomaly Detection with Foreground-Background Disentanglement
Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences（西安光学精密机械研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Xi’an Jiaotong University（西安交通大学）；Zhongnan Hospital of Wuhan University（武汉大学中南医院）
When Semantics Regulate: Rethinking Patch Shuffle and Internal Bias for Generated Image Detection with CLIP
School of CyberSpace Security, Beijing University of Posts and Telecommunications（信息与电子学院，北京邮电大学）；School of Finance, Central University of Finance and Economics（金融学院，中央财经大学）
HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusion
Research Center for Medical AI, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（医学人工智能研究中心，深圳先进技术研究所，中国科学院）；Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（生物医学成像研究中心，深圳先进技术研究所，中国科学院）
NS-Net: Decoupling CLIP Semantic Information through NULL-Space for Generalizable AI-Generated Image Detection
Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology（数字取证工程研究中心，教育部，南京信息科技大学）；University of Macau（澳门大学）
COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Shenzhen Technology University（深圳技术大学）；University of Science and Technology of China（中国科学技术大学）
MOST: Motion Diffusion Model for Rare Text via Temporal Clip Banzhaf Interaction
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学）；Department of Computer Science, University of Durham（计算机科学系，达勒姆大学）
PSMP-CLIP: Patch-Prompt SAM and Multi-Semantic Prompting for CLIP-Based Zero-Shot Anomaly Detection
Harbin Engineering University（哈尔滨工程大学）；Key Laboratory of Advanced Marine Communication and Information Technology（先进海洋通信与信息技术重点实验室）；University of Toronto（多伦多大学）；Kanagawa University（神奈川大学）
Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentation
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Didi Chuxing（滴滴出行）
Probing CLIP's Comprehension of 360-Degree Textual and Visual Semantics
Department of Statistical Science, University College London, UK（伦敦大学统计科学系）；School of Mathematics and Statistics, University of Glasgow, UK（格拉斯哥大学数学与统计学学院）；Department of Computer Science, University of Bath, UK（巴斯大学计算机科学系）
CLIP-SENet: CLIP-based Semantic Enhancement Network for Vehicle Re-identification
School of Computer Science and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机科学与人工智能学院）；Intelligent Transportation Systems Research Center, Wuhan University of Technology（武汉理工大学智能交通系统研究中心）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间安全学院）；School of Automation, Wuhan University of Technology（武汉理工大学自动化学院）
CLIP4VI-ReID: Learning Modality-shared Representations via CLIP Semantic Bridge for Visible-Infrared Person Re-identification
Shandong Key Laboratory of Ubiquitous Intelligent Computing, School of Information Science and Engineering, University of Jinan（山东 ubiquitous 智能计算重点实验室，信息科学与工程学院，济南大学）；College of Information Science and Technology & College of Artificial Intelligence, Nanjing Forestry University（信息科学与技术学院及人工智能学院，南京林业大学）；Computer Systems Engineering Department, Universidad Politécnica de Madrid（计算机系统工程系，马德里理工大学）
Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation
Graduate School of Informatics, Nagoya University（名古屋大学信息学研究科）；AI Thrust, The Hong Kong University of Science and Technology（香港科学与技术大学人工智能研究组）；INSAIT, Sofia University（索菲亚大学INSAIT）
