arXiv 2507.096152025-07-13cs.CV

通过自训练对齐分数实现CLIP的细粒度适配

Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score

Mohamed Bin Zayed University of Artificial Intelligence（莫莫德·本·扎耶德人工智能大学）；IIT Delhi（德里印度理工学院）；Alexandria University（亚历山大大学）

Eman Ali, Sathira Silva, Chetan Arora, Muhammad Haris Khan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对CLIP在细粒度分类无监督适配中固定对齐分数和伪标签策略的局限，提出FAIR方法，利用类别描述锚点动态对齐图文特征并定义学习对齐分数，结合自训练加权机制优化伪标签，在13个数据集上取得2.78%的总体提升。

摘要：

像CLIP这样的视觉语言模型（VLMs）通过对比预训练对齐图像和文本表示，在零样本学习中表现出色。现有针对VLM细粒度分类的无监督适配（UA）方法，要么依赖无法捕捉演变的细微类别差异的固定对齐分数，要么使用限制可扩展性的计算成本高昂的伪标签策略。相比之下，我们展示了在适配过程中建模细粒度跨模态交互，能够产生更准确、更具类别区分度的伪标签，并显著优于最先进（SOTA）方法的性能。我们提出了细粒度对齐与交互优化（FAIR），这是一种创新方法，通过一组类别描述锚点（CDA）将局部图像特征与描述性语言嵌入动态对齐。这使得定义学习对齐分数（LAS）成为可能，它将CDA作为自适应分类器，促进跨模态交互以改善无监督适配中的自训练。此外，我们提出了一种自训练加权机制，旨在类别间存在歧义时优化伪标签。我们的方法FAIR在细粒度无监督适配中带来了显著的性能提升，与SOTA方法相比，在13个细粒度数据集上实现了2.78%的显著总体增益。

英文摘要：

Vision-language models (VLMs) like CLIP excel in zero-shot learning by aligning image and text representations through contrastive pretraining. Existing approaches to unsupervised adaptation (UA) for fine-grained classification with VLMs either rely on fixed alignment scores that cannot capture evolving, subtle class distinctions or use computationally expensive pseudo-labeling strategies that limit scalability. In contrast, we show that modeling fine-grained cross-modal interactions during adaptation produces more accurate, class-discriminative pseudo-labels and substantially improves performance over state-of-the-art (SOTA) methods. We introduce Fine-grained Alignment and Interaction Refinement (FAIR), an innovative approach that dynamically aligns localized image features with descriptive language embeddings through a set of Class Description Anchors (CDA). This enables the definition of a Learned Alignment Score (LAS), which incorporates CDA as an adaptive classifier, facilitating cross-modal interactions to improve self-training in unsupervised adaptation. Furthermore, we propose a self-training weighting mechanism designed to refine pseudo-labels in the presence of inter-class ambiguities. Our approach, FAIR, delivers a substantial performance boost in fine-grained unsupervised adaptation, achieving a notable overall gain of 2.78% across 13 fine-grained datasets compared to SOTA methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。