面向下游任务学习排序预训练视觉-语言模型
Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks
AI总结：
本文提出无监督视觉-语言模型选择问题，并设计视觉-文本图对齐方法VEGA，通过度量视觉与文本图在节点和边级别的相似度，在无标注下游任务上可靠估计VLM性能。
摘要：
视觉语言模型（VLMs）如CLIP在分类基准上展现出卓越的零样本能力。然而，在无标注的下游任务上选择性能最高的VLM并非易事。现有的VLM选择方法聚焦于仅使用类别名称的设置，依赖有监督的大规模数据集和大语言模型，这在部署期间可能无法获取或不可行。本文引入了无监督视觉-语言模型选择问题，即只有无监督下游数据集可用，且不提供任何额外信息。为解决该问题，我们提出了一种称为视觉-文本图对齐（VEGA）的方法，通过测量VLM在下游任务上两种模态之间的对齐程度，在没有任何标注的情况下选择VLM。VEGA的动机源于VLM的预训练范式，该范式将视觉和文本模态中具有相同语义的特征对齐，从而将两种模态映射到共享表示空间。具体而言，我们首先分别在视觉特征和文本特征上构建两个图。然后将VEGA定义为视觉图和文本图在节点级和边级上的整体相似度。在三个不同基准上的大量实验，涵盖多种应用场景和下游数据集，表明VEGA能够持续提供对VLM在无标注下游任务上性能的可靠且准确的估计。
英文摘要：
Vision language models (VLMs) like CLIP show stellar zero-shot capability on classification benchmarks. However, selecting the VLM with the highest performance on the unlabeled downstream task is non-trivial. Existing VLM selection methods focus on the class-name-only setting, relying on a supervised large-scale dataset and large language models, which may not be accessible or feasible during deployment. This paper introduces the problem of \textbf{unsupervised vision-language model selection}, where only unsupervised downstream datasets are available, with no additional information provided. To solve this problem, we propose a method termed Visual-tExtual Graph Alignment (VEGA), to select VLMs without any annotations by measuring the alignment of the VLM between the two modalities on the downstream task. VEGA is motivated by the pretraining paradigm of VLMs, which aligns features with the same semantics from the visual and textual modalities, thereby mapping both modalities into a shared representation space. Specifically, we first construct two graphs on the vision and textual features, respectively. VEGA is then defined as the overall similarity between the visual and textual graphs at both node and edge levels. Extensive experiments across three different benchmarks, covering a variety of application scenarios and downstream datasets, demonstrate that VEGA consistently provides reliable and accurate estimates of VLMs' performance on unlabeled downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
