LOVM: 仅基于语言的视觉模型选择
LOVM: Language-Only Vision Model Selection
AI总结：
针对预训练VLM下游选型需标注数据集、评估成本高的问题，提出仅用下游任务文本描述完成模型选择与性能预测的LOVM任务，并构建含35个预训练VLM、23个数据集的基准。
摘要：
预训练多模态视觉语言模型（VLM）凭借在下游视觉应用中的优异表现，尤其是在少样本和零样本场景下的表现，正变得日益流行。然而，为特定下游应用选择性能最佳的VLM并非易事，因为这一选择依赖于数据集和任务本身。同时，在一个新应用上穷尽评估所有可用的VLM不仅耗费时间与计算资源，还需要收集带标注的评估数据集。随着开源VLM变体数量的增长，亟需一种无需使用整理好的评估数据集的高效模型选择策略。本文提出了一项新任务和基准，用于在无需获取下游任务数据集的情况下高效评估VLM的零样本性能。具体而言，我们提出了LOVM（仅基于语言的视觉模型选择）这一新任务，要求相关方法仅依据目标下游应用的文本描述，同时完成模型选择和性能预测。随后我们构建了一个大规模LOVM基准，包含对35个预训练VLM、23个数据集的真实评估结果，要求相关方法对这些预训练VLM进行排序并预测其零样本性能。
英文摘要：
Pre-trained multi-modal vision-language models (VLMs) are becoming increasingly popular due to their exceptional performance on downstream vision applications, particularly in the few- and zero-shot settings. However, selecting the best-performing VLM for some downstream applications is non-trivial, as it is dataset and task-dependent. Meanwhile, the exhaustive evaluation of all available VLMs on a novel application is not only time and computationally demanding but also necessitates the collection of a labeled dataset for evaluation. As the number of open-source VLM variants increases, there is a need for an efficient model selection strategy that does not require access to a curated evaluation dataset. This paper proposes a novel task and benchmark for efficiently evaluating VLMs' zero-shot performance on downstream applications without access to the downstream task dataset. Specifically, we introduce a new task LOVM: Language-Only Vision Model Selection, where methods are expected to perform both model selection and performance prediction based solely on a text description of the desired downstream application. We then introduced an extensive LOVM benchmark consisting of ground-truth evaluations of 35 pre-trained VLMs and 23 datasets, where methods are expected to rank the pre-trained VLMs and predict their zero-shot performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
