arXiv 2305.180102024-02-21cs.CVcs.MM

基于CLIP奖励的测试时自适应用于视觉-语言模型的零样本泛化

Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models

University of Technology Sydney（悉尼科技大学）；Zhejiang University（浙江大学）；Stanford University（斯坦福大学）；Baidu Inc.（百度公司）

Shuai Zhao, Xiaohan Wang, Linchao Zhu, Yi Yang

备注：: accepted by ICLR 2024, project page at https://mzhaoshuai.github.io/RLCF/, code is at https://github.com/mzhaoshuai/RLCF

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于CLIP反馈的强化学习框架RLCF，在测试时利用CLIP奖励纠正视觉-语言模型输出，提升其在分类、检索和图像描述等任务上的零样本泛化能力。

摘要：

预训练视觉-语言模型（VLMs）在语言监督下学习的一个引人注目的方面是其令人印象深刻的零样本泛化能力。然而，这种能力受到训练数据与测试数据之间分布偏移的阻碍。以往针对零样本分类中VLMs的测试时自适应（TTA）方法依赖于最小化模型输出的熵，往往容易陷入错误的模型预测。在本工作中，我们提出带有反馈的TTA，以纠正模型输出并防止模型变得盲目自信。具体而言，在TTA过程中采用一个CLIP模型作为奖励模型，并为VLM提供反馈。给定单个测试样本，强制VLM最大化输入与从VLM输出分布中采样结果之间的CLIP奖励。所提出的基于CLIP反馈的强化学习（RLCF）框架具有高度的灵活性和通用性。除了分类任务之外，借助任务特定的采样策略和合适的奖励基线选择，RLCF可以轻松扩展到不仅包括检索等判别任务，还包括图像描述等泛化任务，从而提升VLMs的零样本泛化能力。根据这些视觉-语言任务的特点，我们构建了不同的全TTA流程并结合RLCF，以提高各种VLMs的零样本泛化能力。大量实验以及有前景的实证结果证明了RLCF的有效性。代码可在 https://github.com/mzhaoshuai/RLCF 获取。

英文摘要：

One fascinating aspect of pre-trained vision-language models~(VLMs) learning under language supervision is their impressive zero-shot generalization capability. However, this ability is hindered by distribution shifts between the training and testing data. Previous test time adaptation~(TTA) methods for VLMs in zero-shot classification rely on minimizing the entropy of model outputs, tending to be stuck in incorrect model predictions. In this work, we propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confident. Specifically, a CLIP model is adopted as the reward model during TTA and provides feedback for the VLM. Given a single test sample, the VLM is forced to maximize the CLIP reward between the input and sampled results from the VLM output distribution. The proposed \textit{reinforcement learning with CLIP feedback~(RLCF)} framework is highly flexible and universal. Beyond the classification task, with task-specific sampling strategies and a proper reward baseline choice, RLCF can be easily extended to not only discrimination tasks like retrieval but also generalization tasks like image captioning, improving the zero-shot generalization capacity of VLMs. According to the characteristics of these VL tasks, we build different fully TTA pipelines with RLCF to improve the zero-shot generalization ability of various VLMs. Extensive experiments along with promising empirical results demonstrate the effectiveness of RLCF. The code is available at https://github.com/mzhaoshuai/RLCF.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。