面向视觉-语言模型的无监督原型适配器
Unsupervised Prototype Adapter for Vision-Language Models
AI总结：
针对视觉-语言模型有监督微调依赖标注样本、扩展性受限的问题，提出无监督原型适配器UP-Adapter，利用CLIP选高置信样本生成原型初始化可学习模型，残差融合原模型预测，性能显著优于多种少监督及现有无监督方法。
摘要：
近年来，大规模预训练视觉-语言模型（如CLIP和ALIGN）在获取可迁移视觉表征方面展现出显著成效。为利用这些模型中蕴含的宝贵知识服务下游任务，研究者已开发出包括提示调优方法和基于适配器的方法在内的多种微调方案，以在有监督条件下高效适配视觉-语言模型。然而，这些方法依赖标注样本的可用性，而获取标注样本既耗费人力又耗时，从而限制了可扩展性。为解决这一问题，本研究设计了一种面向视觉-语言模型的无监督微调方法，名为无监督原型适配器（Unsupervised Prototype Adapter，UP-Adapter）。具体而言，针对无标注的目标数据集，我们利用CLIP的图文对齐能力，为每个类别自动选取置信度最高的样本。利用这些选定样本，我们生成类别原型，作为可学习原型模型的初始化参数。微调完成后，原型模型的预测结果与原始CLIP的预测结果通过残差连接相结合，用于执行下游识别任务。我们在图像识别和域泛化任务上开展的大量实验结果表明，所提出的无监督方法大幅优于8-shot CoOp、8-shot Tip-Adapter，以及当前最优的UPL方法。
英文摘要：
Recently, large-scale pre-trained vision-language models (e.g. CLIP and ALIGN) have demonstrated remarkable effectiveness in acquiring transferable visual representations. To leverage the valuable knowledge encoded within these models for downstream tasks, several fine-tuning approaches, including prompt tuning methods and adapter-based methods, have been developed to adapt vision-language models effectively with supervision. However, these methods rely on the availability of annotated samples, which can be labor-intensive and time-consuming to acquire, thus limiting scalability. To address this issue, in this work, we design an unsupervised fine-tuning approach for vision-language models called Unsupervised Prototype Adapter (UP-Adapter). Specifically, for the unannotated target datasets, we leverage the text-image aligning capability of CLIP to automatically select the most confident samples for each class. Utilizing these selected samples, we generate class prototypes, which serve as the initialization for the learnable prototype model. After fine-tuning, the prototype model prediction is combined with the original CLIP's prediction by a residual connection to perform downstream recognition tasks. Our extensive experimental results on image recognition and domain generalization show that the proposed unsupervised method outperforms 8-shot CoOp, 8-shot Tip-Adapter, and also the state-of-the-art UPL method by large margins.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
