arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.16977cs.CVcs.MM
收藏

用于少样本 CLIP 的选择性视觉-语言子空间投影

Selective Vision-Language Subspace Projection for Few-shot CLIP

University of Science and Technology of China(中国科学技术大学);Nanyang Technological University(南洋理工大学)

Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, Hanwang Zhang

AI总结:

针对 CLIP 编码特征中存在模态差异导致分类性能受限的问题,提出免训练的选择性视觉-语言子空间投影(SSP)方法,利用局部图像特征构建子空间并设计双投影模块实现图文对齐,在 11 个数据集上优于现有 SOTA。

摘要:

诸如 CLIP 的视觉-语言模型能够将不同模态数据映射到统一的特征空间中,通过测量给定图像和文本的相似度来实现零样本/少样本推理。然而,大多数现有方法忽略了 CLIP 编码特征中的模态差异,表现为文本和图像特征彼此相距甚远,导致分类性能受限。为了解决这个问题,我们引入了一种名为选择性视觉-语言子空间投影(SSP)的方法,该方法结合局部图像特征并将其作为桥梁来增强图像-文本对之间的对齐。具体而言,我们的 SSP 框架包含两个并行模块:视觉投影器和语言投影器。两个投影器利用局部图像特征分别为图像和文本张成各自的子空间,从而将图像和文本特征投影到各自的子空间中实现特征对齐。此外,我们的方法仅涉及免训练的矩阵计算,并且可以无缝集成到先进的基于 CLIP 的少样本学习框架中。在 11 个数据集上的大量实验证明了 SSP 卓越的文本-图像对齐能力,优于当前最先进(SOTA)的对齐方法。代码可在 https://github.com/zhuhsingyuu/SSP 获取。

英文摘要:

Vision-language models such as CLIP are capable of mapping the different modality data into a unified feature space, enabling zero/few-shot inference by measuring the similarity of given images and texts. However, most existing methods overlook modality gaps in CLIP's encoded features, which is shown as the text and image features lie far apart from each other, resulting in limited classification performance. To tackle this issue, we introduce a method called Selective Vision-Language Subspace Projection (SSP), which incorporates local image features and utilizes them as a bridge to enhance the alignment between image-text pairs. Specifically, our SSP framework comprises two parallel modules: a vision projector and a language projector. Both projectors utilize local image features to span the respective subspaces for image and texts, thereby projecting the image and text features into their respective subspaces to achieve alignment. Moreover, our approach entails only training-free matrix calculations and can be seamlessly integrated into advanced CLIP-based few-shot learning frameworks. Extensive experiments on 11 datasets have demonstrated SSP's superior text-image alignment capabilities, outperforming the state-of-the-art alignment methods. The code is available at https://github.com/zhuhsingyuu/SSP

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑