Vision-language models such as CLIP are capable of mapping the different modality data into a unified feature space, enabling zero/few-shot inference by measuring the similarity of given images and texts. However, most existing methods overlook modality gaps in CLIP's encoded features, which is shown as the text and image features lie far apart from each other, resulting in limited classification performance. To tackle this issue, we introduce a method called Selective Vision-Language Subspace Projection (SSP), which incorporates local image features and utilizes them as a bridge to enhance the alignment between image-text pairs. Specifically, our SSP framework comprises two parallel modules: a vision projector and a language projector. Both projectors utilize local image features to span the respective subspaces for image and texts, thereby projecting the image and text features into their respective subspaces to achieve alignment. Moreover, our approach entails only training-free matrix calculations and can be seamlessly integrated into advanced CLIP-based few-shot learning frameworks. Extensive experiments on 11 datasets have demonstrated SSP's superior text-image alignment capabilities, outperforming the state-of-the-art alignment methods. The code is available at https://github.com/zhuhsingyuu/SSP
Parts of Speech-Grounded Subspaces in Vision-Language Models
Queen Mary University of London(伦敦玛丽女王大学);National and Kapodistrian University of Athens(雅典国立卡波季斯特里安大学);Archimedes/Athena RC(阿基米德/雅典研究与创新中心);The Cyprus Institute(塞浦路斯研究所)
IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
Media Integration and Communication Center (MICC), University of Florence, Italy(意大利佛罗伦萨大学媒体集成与通信中心);Department of Computer Science, Universitat Autònoma de Barcelona, Spain(西班牙巴塞罗那自治大学计算机科学系);Computer Vision Center, Barcelona, Spain(西班牙巴塞罗那计算机视觉中心);IDEAS Research Institute, Warsaw, Poland(波兰华沙IDEAS研究所)
Project and Mix: Task-Semantic Prototypes for Few-Shot Image Classification
Department of Computer Science, Universitat Autònoma de Barcelona, Spain(巴塞罗那自治大学计算机科学系);Computer Vision Center, Barcelona, Spain(巴塞罗那计算机视觉中心);Media Integration and Communication Center, University of Florence, Italy(佛罗伦萨大学媒体整合与传播中心);Bernoulli Institute, University of Groningen, the Netherlands(格罗宁根大学伯努利学院);IDEAS Research Institute, Poland(波兰IDEAS研究所);ESAT-PSI, KU Leuven, Belgium(比利时KU莱顿大学ESAT-PSI)
Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP
Fraunhofer Heinrich Hertz Institute(弗劳恩霍夫海因里希·赫兹研究所);University of Oxford(牛津大学);Technological University Dublin(都柏林技术大学);Technische Universität Berlin(柏林技术大学)
HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing Jiaotong University(北京交通大学)
Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs
Macquarie University(麦考瑞大学);York University(约克大学);Northern Illinois University(北伊利诺伊大学);University of Technology Sydney(悉尼科技大学);North South University(北南大学);Lancaster University(兰卡斯特大学);Australian National University(澳大利亚国立大学)
When Semantics Regulate: Rethinking Patch Shuffle and Internal Bias for Generated Image Detection with CLIP
School of CyberSpace Security, Beijing University of Posts and Telecommunications(信息与电子学院,北京邮电大学);School of Finance, Central University of Finance and Economics(金融学院,中央财经大学)
Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation
Graduate School of Informatics, Nagoya University(名古屋大学信息学研究科);AI Thrust, The Hong Kong University of Science and Technology(香港科学与技术大学人工智能研究组);INSAIT, Sofia University(索菲亚大学INSAIT)
CLIP-PAE: Projection-Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable, and Controllable Text-Guided Face Manipulation
Low-rank Orthogonal Subspace Intervention for Generalizable Face Forgery Detection
Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology(数字取证工程研究中心,教育部,南京信息科技大学)
Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation
School of Engineering Science, Simon Fraser University(西蒙弗雷泽大学工程科学学院);School of Information Science and Technology, Eastern Institute of Technology(东方理工学院信息科学与技术学院)
SCING:Towards More Efficient and Robust Person Re-Identification through Selective Cross-modal Prompt Tuning
Huazhong University of Science and Technology(华中科技大学);Huazhong Agricultural University(华中农业大学);University of California, Santa Cruz(加州大学圣克ruz分校);City University of Hong Kong (Dongguan)(香港城市大学(东莞))
TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection
Zhejiang University(浙江大学);Tencent YouTu Lab(腾讯YouTu实验室);Singapore Management University(新加坡管理学院);Qilu University of Technology (Shandong Academy of Sciences)(青岛科技大学(山东科学院))