IsoCLIP: 分解CLIP投影器以实现高效的模态内对齐
IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
AI总结：
本文通过分析CLIP投影器的谱特性，发现模态间对齐子空间和各向异性方向，提出无训练方法IsoCLIP去除各向异性方向以改善模态内对齐，在模态内检索和分类任务上降低延迟并超越现有方法。
摘要：
视觉-语言模型如CLIP被广泛用于涉及视觉和文本模态的跨模态任务。然而，当个体模态编码器应用于固有的模态内任务（如图像到图像检索）时，其性能因模态内错位而受损。本文研究CLIP中的模态内错位，重点关注将投影前图像和文本嵌入映射到共享嵌入空间的投影器的作用。通过分析应用于投影特征的余弦相似度形式及其与对比CLIP损失的交互，我们发现在训练期间存在一个负责对齐两种模态的跨模态算子，以及第二个仅强制执行模态内归一化但不促进模态内对齐的模态内算子。通过对跨模态算子的谱分析，我们识别出一个近似各向同性的子空间，其中两种模态良好对齐，以及每个模态特有的各向异性方向。我们证明该对齐子空间可以直接从投影器权重中获得，并且去除各向异性方向可改善模态内对齐。我们在模态内检索和分类基准上的实验表明，我们的无训练方法减少了模态内错位，大大降低了延迟，并在多个预训练的类CLIP模型上优于现有方法。代码公开于：https://github.com/simomagi/IsoCLIP。
英文摘要：
Vision-Language Models like CLIP are extensively used for inter-modal tasks which involve both visual and text modalities. However, when the individual modality encoders are applied to inherently intra-modal tasks like image-to-image retrieval, their performance suffers from the intra-modal misalignment. In this paper we study intra-modal misalignment in CLIP with a focus on the role of the projectors that map pre-projection image and text embeddings into the shared embedding space. By analyzing the form of the cosine similarity applied to projected features, and its interaction with the contrastive CLIP loss, we show that there is an inter-modal operator responsible for aligning the two modalities during training, and a second, intra-modal operator that only enforces intra-modal normalization but does nothing to promote intra-modal alignment. Via spectral analysis of the inter-modal operator, we identify an approximately isotropic subspace in which the two modalities are well-aligned, as well as anisotropic directions specific to each modality. We demonstrate that this aligned subspace can be directly obtained from the projector weights and that removing the anisotropic directions improves intra-modal alignment. Our experiments on intra-modal retrieval and classification benchmarks show that our training-free method reduces intra-modal misalignment, greatly lowers latency, and outperforms existing approaches across multiple pre-trained CLIP-like models. The code is publicly available at: https://github.com/simomagi/IsoCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
