基于锚点的视觉-语言模型鲁棒微调
Anchor-based Robust Finetuning of Vision-Language Models
AI总结：
针对视觉-语言模型微调后OOD泛化下降的问题，本文提出文本补偿锚点和图文对锚点提供辅助语义监督，在保持分布内性能的同时刷新域迁移与零样本基准。
摘要：
我们的目标是在不损害视觉-语言模型分布外（OOD）泛化能力的情况下对其进行微调。我们处理两类OOD泛化，即：i）域迁移，例如从自然图像到素描图像；以及 ii）识别微调数据中未包含类别的零样本能力。可以认为，微调后OOD泛化能力下降源于过度简化的微调目标，该目标仅提供类别信息，例如“一张[CLASS]的照片”。这不同于CLIP预训练过程，后者具有丰富语义信息的大量文本监督。因此，我们提出利用具有丰富语义信息的辅助监督来补偿微调过程，这些监督充当保持OOD泛化能力的锚点。具体而言，我们的方法中精心设计了两类锚点，包括：i）文本补偿锚点，它使用来自微调集的图像，但通过预训练描述器丰富文本监督；ii）图文对锚点，它根据下游任务从类似CLIP预训练数据的数据集中检索得到，并与具有丰富语义的原始CLIP文本相关联。这些锚点被用作辅助语义信息，以维持CLIP的原始特征空间，从而保持OOD泛化能力。综合实验表明，我们的方法取得了与传统微调相近的分布内性能，同时在域迁移和零样本学习基准上获得了新的最先进结果。
英文摘要：
We aim at finetuning a vision-language model without hurting its out-of-distribution (OOD) generalization. We address two types of OOD generalization, i.e., i) domain shift such as natural to sketch images, and ii) zero-shot capability to recognize the category that was not contained in the finetune data. Arguably, the diminished OOD generalization after finetuning stems from the excessively simplified finetuning target, which only provides the class information, such as ``a photo of a [CLASS]''. This is distinct from the process in that CLIP was pretrained, where there is abundant text supervision with rich semantic information. Therefore, we propose to compensate for the finetune process using auxiliary supervision with rich semantic information, which acts as anchors to preserve the OOD generalization. Specifically, two types of anchors are elaborated in our method, including i) text-compensated anchor which uses the images from the finetune set but enriches the text supervision from a pretrained captioner, ii) image-text-pair anchor which is retrieved from the dataset similar to pretraining data of CLIP according to the downstream task, associating with the original CLIP text with rich semantics. Those anchors are utilized as auxiliary semantic information to maintain the original feature space of CLIP, thereby preserving the OOD generalization capabilities. Comprehensive experiments demonstrate that our method achieves in-distribution performance akin to conventional finetuning while attaining new state-of-the-art results on domain shift and zero-shot learning benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
