弥合视觉语言模型选择中的模态与能力差距
Bridge the Modality and Capability Gaps in Vision-Language Model Selection
AI总结：
针对纯语言VLM选择中的模态与能力差距问题，提出基于最优传输的SWAB方法，通过迁移开源数据集统计量弥合差距，无需目标数据集图像即可准确预测VLM性能排名，经多组实验验证有效。
摘要：
视觉语言模型（Vision Language Models, VLMs）通过将图像与文本类别名称配对，在零样本图像分类任务中表现出色。预训练VLM的种类不断增多，提升了为特定任务找到合适VLM的可能性。为了更好地复用VLM资源并充分发挥其在不同零样本图像分类任务中的潜力，一种颇具前景的策略是：仅依靠目标数据集的文本数据，无需访问数据集的图像，从VLM库（VLM Zoo）中选择合适的预训练VLM。\n本文分析了这种纯语言VLM选择任务中评估VLM能力时存在的两个固有挑战：一是“模态差距（Modality Gap）”——即VLM的嵌入在两种不同模态间存在差异，导致文本无法可靠替代图像；二是“能力差距（Capability Gap）”——即VLM的整体排名与其在目标数据集上的排名存在差异，阻碍了从模型的通用性能直接预测其数据集特定性能。\n我们提出了带差距弥合的VLM选择方法（VLM Selection With gAp Bridging, SWAB），以缓解这两种差距带来的负面影响。SWAB首先采用最优传输方法，通过传输矩阵捕捉开源数据集与目标数据集之间的相关性；随后利用该矩阵将开源数据集中VLM的有效统计量迁移到目标数据集，以此弥合两种差距。通过弥合两类差距以获得更好的测试图像替代表征，SWAB无需数据集图像即可准确预测不同VLM在目标任务上的性能排名。在多种VLM和图像分类数据集上开展的实验验证了SWAB的有效性。
英文摘要：
Vision Language Models (VLMs) excel in zero-shot image classification by pairing images with textual category names. The expanding variety of Pre-Trained VLMs enhances the likelihood of identifying a suitable VLM for specific tasks. To better reuse the VLM resource and fully leverage its potential on different zero-shot image classification tasks, a promising strategy is selecting appropriate Pre-Trained VLMs from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset's images. In this paper, we analyze two inherent challenges in assessing the ability of a VLM in this Language-Only VLM selection: the "Modality Gap" - the disparity in VLM's embeddings across two different modalities, making text a less reliable substitute for images; and the "Capability Gap" - the discrepancy between the VLM's overall ranking and its ranking for target dataset, hindering direct prediction of a model's dataset-specific performance from its general performance. We propose VLM Selection With gAp Bridging (SWAB) to mitigate the negative impact of two gaps. SWAB first adopts optimal transport to capture the relevance between open-source and target datasets with a transportation matrix. It then uses this matrix to transfer useful statistics of VLMs from open-source datasets to the target dataset for bridging two gaps. By bridging two gaps to obtain better substitutes for test images, SWAB can accurately predict the performance ranking of different VLMs on the target task without the need for the dataset's images. Experiments across various VLMs and image classification datasets validate SWAB's effectiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
