arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.02243cs.CV
收藏

检索增强的视觉提示学习用于少样本分类

Retrieval-Enhanced Visual Prompt Learning for Few-shot Classification

College of Information Engineering, Zhejiang University of Technology(浙江工业大学信息工程学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Computer Science, The University of Adelaide(阿德莱德大学计算机学院)

Jintao Rong, Hao Chen, Linlin Ou, Tianxiao Chen, Xinyi Yu, Yifan Liu

AI总结:

针对少样本细粒度分类中领域差距大的问题,提出检索增强的视觉提示学习方法 RePrompt,通过缓存并重用下游任务知识来增强提示学习,在多个视觉数据集上取得最先进性能。

摘要:

对比语言-图像预训练(CLIP)模型已被广泛应用于各种下游视觉任务。少样本学习范式已被广泛采用以增强其在这些任务上的能力。然而,当前的范式可能在细粒度分类(如卫星图像识别)上遇到困难,因为领域差距不断扩大。为了解决这一局限性,我们提出了检索增强的视觉提示学习(RePrompt),该方法引入检索机制来缓存和重用下游任务的知识。RePrompt 从训练样本或外部数据(如果可用)构建检索数据库,并使用检索机制来增强一个简单提示学习基线的多个阶段,从而缩小领域差距。在推理过程中,我们增强的模型可以参考检索带来的相似样本,以做出更准确的预测。详细分析表明,检索有助于改善后期特征的分布,从而提高下游任务的泛化能力。RePrompt 在广泛的视觉数据集上取得了最先进的性能,包括 11 个图像数据集、3 个视频数据集、1 个多视图数据集和 4 个领域泛化基准。

英文摘要:

The Contrastive Language-Image Pretraining (CLIP) model has been widely used in various downstream vision tasks. The few-shot learning paradigm has been widely adopted to augment its capacity for these tasks. However, current paradigms may struggle with fine-grained classification, such as satellite image recognition, due to widening domain gaps. To address this limitation, we propose retrieval-enhanced visual prompt learning (RePrompt), which introduces retrieval mechanisms to cache and reuse the knowledge of downstream tasks. RePrompt constructs a retrieval database from either training examples or external data if available, and uses a retrieval mechanism to enhance multiple stages of a simple prompt learning baseline, thus narrowing the domain gap. During inference, our enhanced model can reference similar samples brought by retrieval to make more accurate predictions. A detailed analysis reveals that retrieval helps to improve the distribution of late features, thus, improving generalization for downstream tasks. Reprompt attains state-of-the-art performance on a wide range of vision datasets, including 11 image datasets, 3 video datasets, 1 multi-view dataset, and 4 domain generalization benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑