arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.04087cs.CVcs.AIcs.LG
收藏

一种难以超越的基线：无需训练的CLIP自适应方法

A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation

University of Science and Technology of China（中国科学技术大学）；CRIPAC & MAIS, Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所CRIPAC与MAIS）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Nanjing University（南京大学）

Zhengbo Wang, Jian Liang, Lijun Sheng, Ran He, Zilei Wang, Tieniu Tan

备注：
Accepted by ICLR 2024
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种无需训练的CLIP自适应方法，通过高斯判别分析估计类别参数并与零样本分类器集成，在17个数据集上达到或超越现有微调方法，适用于少样本、不平衡及分布外场景。

摘要：

对比语言-图像预训练（CLIP）因其显著的零样本能力而广受欢迎。近期研究聚焦于开发高效的微调方法，如提示学习和适配器，以提升CLIP在下游任务中的性能。然而，这些方法仍需要额外的训练时间和计算资源，这对于资源受限的设备而言并不理想。在本文中，我们重新审视一种经典算法——高斯判别分析（GDA），并将其应用于CLIP的下游分类任务。通常，GDA假设每个类别的特征遵循具有相同协方差的高斯分布。通过利用贝叶斯公式，分类器可以用类别均值和协方差来表示，这些参数可以直接从数据中估计，无需训练。为了整合视觉和文本两种模态的知识，我们将其与CLIP内部的原始零样本分类器进行集成。在17个数据集上的大量实验结果验证了，我们的方法在少样本分类、不平衡学习和分布外泛化方面，优于或达到了与最先进方法相当的性能。此外，我们将方法扩展到基础到新类别的泛化和无监督学习，再次证明了其相对于竞争方法的优越性。我们的代码公开在\url{https://github.com/mrflogs/ICLR24}。

英文摘要：

Contrastive Language-Image Pretraining (CLIP) has gained popularity for its remarkable zero-shot capacity. Recent research has focused on developing efficient fine-tuning methods, such as prompt learning and adapter, to enhance CLIP's performance in downstream tasks. However, these methods still require additional training time and computational resources, which is undesirable for devices with limited resources. In this paper, we revisit a classical algorithm, Gaussian Discriminant Analysis (GDA), and apply it to the downstream classification of CLIP. Typically, GDA assumes that features of each class follow Gaussian distributions with identical covariance. By leveraging Bayes' formula, the classifier can be expressed in terms of the class means and covariance, which can be estimated from the data without the need for training. To integrate knowledge from both visual and textual modalities, we ensemble it with the original zero-shot classifier within CLIP. Extensive results on 17 datasets validate that our method surpasses or achieves comparable results with state-of-the-art methods on few-shot classification, imbalanced learning, and out-of-distribution generalization. In addition, we extend our method to base-to-new generalization and unsupervised learning, once again demonstrating its superiority over competing approaches. Our code is publicly available at \url{https://github.com/mrflogs/ICLR24}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑