arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.16486cs.CV
收藏

通过测试时提示调优使视觉语言模型适应开放类别

Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)

Zhengqing Gao, Xiang Ao, Xu-Yao Zhang, Cheng-Lin Liu

AI总结:

针对视觉语言模型在开放集适应中学习提示泛化能力差的问题,提出测试时提示调优方法,利用最大概念匹配分数作为动态权重为每张图像生成输入条件提示,在11个数据集上平均优于所有比较方法。

摘要:

将预训练模型适应到开放类别是机器学习中的一个具有挑战性的问题。视觉语言模型充分利用文本模态的知识,展现出强大的零样本识别性能,这自然适合各种开放集问题。最近,一些研究专注于将此类模型微调到下游任务。提示调优方法通过在少样本数据上学习上下文向量取得了巨大的改进。然而,通过在包含新类别的测试数据的开放集适应设置下进行评估,我们发现存在一个困境,即学习到的提示比手工制作的提示具有更差的泛化能力。在本文中,我们考虑结合两者的优点,并提出了一种测试时提示调优方法,该方法利用最大概念匹配(MCM)分数作为动态权重,在测试期间为每张图像生成一个输入条件提示。通过在11个不同数据集上的广泛实验,我们表明,综合考虑基类和新类,我们提出的方法平均优于所有比较方法。代码可在 https://github.com/gaozhengqing/TTPT 获取。

英文摘要:

Adapting pre-trained models to open classes is a challenging problem in machine learning. Vision-language models fully explore the knowledge of text modality, demonstrating strong zero-shot recognition performance, which is naturally suited for various open-set problems. More recently, some research focuses on fine-tuning such models to downstream tasks. Prompt tuning methods achieved huge improvements by learning context vectors on few-shot data. However, through the evaluation under open-set adaptation setting with the test data including new classes, we find that there exists a dilemma that learned prompts have worse generalization abilities than hand-crafted prompts. In this paper, we consider combining the advantages of both and come up with a test-time prompt tuning approach, which leverages the maximum concept matching (MCM) scores as dynamic weights to generate an input-conditioned prompt for each image during test. Through extensive experiments on 11 different datasets, we show that our proposed method outperforms all comparison methods on average considering both base and new classes. The code is available at https://github.com/gaozhengqing/TTPT

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑