arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.06126cs.CV
收藏

面向冻结视觉-语言模型测试时视觉识别的上下文提示学习

In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model

Beijing University of Posts and Telecommunications(北京邮电大学);Baidu Inc.(百度公司);Swansea University(斯旺西大学)

Junhui Yin, Xinyu Zhang, Lin Wu, Xiaojie Wang

AI总结:

针对测试分布偏移下CLIP等预训练视觉语言模型性能下降问题,提出测试时上下文提示学习方法InCPL,通过少量标注样本作上下文、语言到视觉转换、上下文感知无监督损失及循环学习,实现冻结模型的测试时自适应,性能达SOTA。

摘要:

当前预训练视觉-语言模型(如CLIP)已在各类下游任务中展现出卓越的零样本泛化能力,但当测试输入呈现不同分布时,其性能会显著下降。本文探究了测试时提示调优(TTPT)的概念,该方法通过仅涉及测试样本的单步无监督优化,助力CLIP模型适配全新的下游任务。受自然语言处理(NLP)中上下文学习的启发,我们提出用于测试时视觉识别任务的上下文提示学习(InCPL),为预训练视觉-语言模型赋予带标注示例作为下游任务的上下文信息。具体而言,InCPL将新测试样本与极少的带标注示例(有时仅一个)关联作为上下文信息,实现对测试样本的可靠标签估计并促进模型适配。为达成这一目标,InCPL采用高效的语言到视觉转换器,挖掘文本先验信息以用于视觉提示学习。此外,我们引入了一种上下文感知的无监督损失,以优化针对测试样本定制的视觉提示。最后,我们设计了视觉与文本提示的循环学习策略,确保不同模态间的相互协同。这使得预训练的冻结CLIP模型能够利用其学习到的自适应提示适配任意任务。我们的方法展现出优越的性能,在各类下游数据集上取得了当前最优结果。

英文摘要:

Current pre-trained vision-language models, such as CLIP, have demonstrated remarkable zero-shot generalization capabilities across various downstream tasks. However, their performance significantly degrades when test inputs exhibit different distributions. In this paper, we explore the concept of test-time prompt tuning (TTPT), which facilitates the adaptation of the CLIP model to novel downstream tasks through a one-step unsupervised optimization that involves only test samples. Inspired by in-context learning in natural language processing (NLP), we propose In-Context Prompt Learning (InCPL) for test-time visual recognition tasks, which empowers a pre-trained vision-language model with labeled examples as context information on downstream task. Specifically, InCPL associates a new test sample with very few labeled examples (sometimes just one) as context information, enabling reliable label estimation for the test sample and facilitating model adaptation. To achieve this, InCPL employs an efficient language-to-vision translator to explore the textual prior information for visual prompt learning. Further, we introduce a context-aware unsupervised loss to optimize visual prompts tailored to test samples. Finally, we design a cyclic learning strategy for visual and textual prompts to ensure mutual synergy across different modalities. This enables a pre-trained, frozen CLIP model to adapt to any task using its learned adaptive prompt. Our method demonstrates superior performance and achieves state-of-the-art results across various downstream datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑