CLIP与DINO相遇：利用无标签图像集合微调零样本分类器
CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collections
AI总结：
本文提出无标签提示微调框架NoLA，结合DINO的视觉特征与LLM的文本知识，通过生成文本嵌入、训练对齐模块并微调CLIP视觉编码器，在11个数据集上超越LaFTer平均3.6%。
摘要：
在基础模型时代，CLIP已成为将文本与视觉模态对齐到共同嵌入空间的强大工具。然而，用于训练CLIP的对齐目标往往导致其在细粒度任务上的视觉特征表现欠佳。相比之下，像DINO这样的自监督预训练模型因其专门的训练范式而擅长提取丰富的视觉特征。但是，这些自监督模型需要额外的有监督线性探测步骤，该步骤依赖完全标注的数据，而大规模获取此类数据通常昂贵且困难。在本文中，我们提出了一种无标签提示微调方法，利用自监督学习模型（DINO）的丰富视觉特征和大语言模型（LLM）的广泛文本知识，在使用无标签图像的情况下大幅提升基于CLIP的图像分类性能。我们的方法分三个关键步骤展开：（1）通过利用LLM生成的类别特定描述，我们生成更准确表示对象类别的鲁棒文本特征嵌入，与CLIP默认的基于名称的提示相比，能够实现更有效的零样本分类。（2）随后使用这些文本嵌入生成伪标签，以训练一个对齐模块，该模块整合了基于LLM描述的文本嵌入与DINO视觉特征的互补优势。（3）最后，我们利用训练好的对齐模块，通过DINO辅助的监督对CLIP的视觉编码器进行提示微调。这一三步过程使我们能够充分利用视觉与文本基础模型的优势，形成一种强大而高效的方法，超越了最先进的无标签分类方法。值得注意的是，我们的框架NoLA（No Labels Attached，即“无附加标签”）在11个不同的图像分类数据集上相比最先进的LaFTer取得了平均3.6%的绝对增益。我们的代码和模型可在https://github.com/fazliimam/NoLA找到。
英文摘要：
In the era of foundation models, CLIP has emerged as a powerful tool for aligning text & visual modalities into a common embedding space. However, the alignment objective used to train CLIP often results in subpar visual features for fine-grained tasks. In contrast, SSL-pretrained models like DINO excel at extracting rich visual features due to their specialized training paradigm. Yet, these SSL models require an additional supervised linear probing step, which relies on fully labeled data which is often expensive and difficult to obtain at scale. In this paper, we propose a label-free prompt-tuning method that leverages the rich visual features of self-supervised learning models (DINO) and the broad textual knowledge of large language models (LLMs) to largely enhance CLIP-based image classification performance using unlabeled images. Our approach unfolds in three key steps: (1) We generate robust textual feature embeddings that more accurately represent object classes by leveraging class-specific descriptions from LLMs, enabling more effective zero-shot classification compared to CLIP's default name-specific prompts. (2) These textual embeddings are then used to produce pseudo-labels to train an alignment module that integrates the complementary strengths of LLM description-based textual embeddings & DINO's visual features. (3) Finally, we prompt-tune CLIP's vision encoder through DINO-assisted supervision using the trained alignment module. This three-step process allows us to harness the best of visual & textual foundation models, resulting in a powerful and efficient approach that surpasses state-of-the-art label-free classification methods. Notably, our framework, NoLA (No Labels Attached), achieves an average absolute gain of 3.6% over the state-of-the-art LaFTer across 11 diverse image classification datasets. Our code & models can be found at https://github.com/fazliimam/NoLA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
