arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.18271cs.LGcs.AI
收藏

面向下游任务适配的视觉-语言模型选择与复用

Vision-Language Model Selection and Reuse for Downstream Adaptation

School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)

Hao-Zhe Tan, Zhi Zhou, Yu-Feng Li, Lan-Zhe Guo

AI总结:

本文提出模型标签学习(MLL)范式,通过模型标注、选择与复用三个模块高效地为下游任务挑选并集成视觉-语言模型,并在包含49个VLM和17个数据集的新基准上验证了其有效性。

摘要:

预训练视觉-语言模型(VLMs)在各种视觉任务中日益流行,并且已有多个开源的VLM变体发布。然而,为特定下游任务选择性能最佳的预训练VLM具有挑战性,因为没有任何单一VLM能够在所有下游任务上都取得理想性能,同时由于时间和数据限制,评估所有可用VLM也不可行。为了解决这一问题,本文提出了一种用于下游任务VLM选择与复用的新范式,称为模型标签学习(Model Label Learning,MLL)。该方案包含三个关键模块:模型标注(model labeling),为每个VLM分配标签以描述其专长和效用;模型选择(model selection),将目标任务的需求与模型标签进行匹配;模型复用(model reuse),以集成方式将选定的VLM应用于目标任务。该方案具有很高的计算效率和可成长性,因为模型标注过程独立于目标任务完成,并且其能力可随候选VLM数量的增加而增长。我们还引入了一个用于评估VLM选择方法的新基准,包含49个VLM和17个目标任务数据集。实验结果清楚地证明了所提方法在VLM选择与复用方面的有效性。

英文摘要:

Pre-trained Vision-Language Models (VLMs) are becoming increasingly popular across various visual tasks, and several open-sourced VLM variants have been released. However, selecting the best-performing pre-trained VLM for a specific downstream task is challenging since no single VLM can achieve promising performance on all downstream tasks, and evaluating all available VLMs is impossible due to time and data limitations. To address this problem, this paper proposes a novel paradigm to select and reuse VLM for downstream tasks, called Model Label Learning (MLL). The proposal contains three key modules: \emph{model labeling}, which assigns labels to each VLM to describe their specialty and utility; \emph{model selection}, which matches the requirements of the target task with model labels; and \emph{model reuse}, which applies selected VLMs to the target task in an ensemble manner. The proposal is highly computationally efficient and growable since the model labeling process is completed target task independent and the ability could grow with the number of candidate VLMs. We also introduce a new benchmark for evaluating VLM selection methods, including 49 VLMs and 17 target task datasets. Experimental results clearly demonstrate the effectiveness of the proposed method for selecting and reusing VLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑