arXiv 2406.138752024-06-24cs.CV

WATT：CLIP的权重平均测试时适应

WATT: Weight Average Test-Time Adaptation of CLIP

ÉTS Montréal（蒙特利尔工程学院）；MILA（蒙特利尔学习算法研究所）；CNRS（法国国家科学研究中心）；Université Paris-Saclay（巴黎萨克雷大学）；CentraleSupélec（巴黎中央理工-高等电力学院）

David Osowiechi, Mehrdad Noori, Gustavo Adolfo Vargas Hakim, Moslem Yazdanpanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani, Farzad Beizaee, Ismail Ben Aye… 展开作者

David Osowiechi, Mehrdad Noori, Gustavo Adolfo Vargas Hakim, Moslem Yazdanpanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani, Farzad Beizaee, Ismail Ben Ayed, Christian Desrosiers

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对CLIP在域偏移下泛化能力下降的问题，提出权重平均测试时适应方法WATT，通过多样化文本模板、伪标签更新与权重平均及文本集成策略，无需额外模块即可在多数据集上显著提升性能，且仅需单张图像即可运行。

摘要：

诸如CLIP之类的视觉语言模型（VLMs）在零样本图像分类中取得了前所未有的性能，然而当面临域偏移时，其泛化能力仍可能受到严重挑战。作为回应，我们提出了CLIP的权重平均测试时适应（WATT），这是一种促进该VLM完全测试时适应（TTA）的先驱方法。我们的方法为文本提示采用了一组多样化的模板，增强了CLIP的现有框架。预测结果被用作模型更新的伪标签，随后进行权重平均以全局巩固所学习的信息。此外，我们引入了一种文本集成策略，通过聚合多样的文本线索来提升整体测试性能。我们的发现强调了WATT在提升包括CIFAR-10-C、CIFAR-10.1、CIFAR-100-C、VisDA-C及其他几个具有挑战性的数据集在内的多样化数据集性能方面的有效性，有效覆盖了广泛的域偏移。值得注意的是，这些提升无需额外的模型变换或可训练模块即可实现。此外，与其他测试时适应方法相比，我们的方法仅需单张图像即可有效运行。本研究凸显了创新测试时策略的潜力，强调了其在增强VLMs适应性方面的作用。实现代码可在：https://github.com/Mehrdad-Noori/WATT.git 获取。

英文摘要：

Vision-Language Models (VLMs) such as CLIP have yielded unprecedented performance for zero-shot image classification, yet their generalization capability may still be seriously challenged when confronted to domain shifts. In response, we present Weight Average Test-Time Adaptation (WATT) of CLIP, a pioneering approach facilitating full test-time adaptation (TTA) of this VLM. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Predictions are utilized as pseudo labels for model updates, followed by weight averaging to consolidate the learned information globally. Furthermore, we introduce a text ensemble strategy, enhancing overall test performance by aggregating diverse textual cues. Our findings underscore the efficacy of WATT in enhancing performance across diverse datasets, including CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, VisDA-C, and several other challenging datasets, effectively covering a wide range of domain shifts. Notably, these enhancements are achieved without necessitating additional model transformations or trainable modules. Moreover, compared to other Test-Time Adaptation methods, our approach can operate effectively with just a single image. Highlighting the potential of innovative test-time strategies, this research emphasizes their role in fortifying the adaptability of VLMs. The implementation is available at: \url{https://github.com/Mehrdad-Noori/WATT.git}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。