即插即用的类感知知识注入用于具有视觉-语言模型的提示学习
Plug-and-play Class-aware Knowledge Injection for Prompt Learning with Visual-Language Model
AI总结:
本文提出CAKI框架,通过类特定提示生成和查询-键提示匹配,补充现有方法中的类特定知识,提升基类和新类的性能。
摘要:
提示学习已成为增强视觉-语言模型(VLMs)如CLIP等的有效且广泛应用的技术,特别是在特定领域的零样本分类任务中。现有方法通常专注于学习给定领域的类共享提示或通过条件提示学习生成实例特定提示。尽管这些方法取得了有希望的性能,但它们往往忽略了提示设计中的类特定知识,导致次优结果。根本原因在于:1)类特定提示相比粗粒度的类共享提示提供更细粒度的监督,有助于防止不同类别的数据被错误归类到单一类别;2)与类特定提示相比,实例特定提示忽略了多个实例中的丰富类级信息,可能导致同类别数据被分割到多个类别。为了有效补充现有方法中的类特定知识,我们提出即插即用的类感知知识注入(CAKI)框架。CAKI包含两个关键组件,即类特定提示生成和查询-键提示匹配。前者将类特定知识编码到属于同一类别的少量样本的提示中,并将学习到的提示存储在类级知识库中。后者为每个测试实例提供即插即用的机制,从知识库中检索相关类级知识并注入此类知识以优化模型预测。广泛的实验表明,我们的CAKI有效提高了现有方法在基类和新类上的性能。代码可在https://github.com/yjh576/CAKI公开获取。
英文摘要:
Prompt learning has become an effective and widely used technique in enhancing vision-language models (VLMs) such as CLIP for various downstream tasks, particularly in zero-shot classification within specific domains. Existing methods typically focus on either learning class-shared prompts for a given domain or generating instance-specific prompts through conditional prompt learning. While these methods have achieved promising performance, they often overlook class-specific knowledge in prompt design, leading to suboptimal outcomes. The underlying reasons are: 1) class-specific prompts offer more fine-grained supervision compared to coarse class-shared prompts, which helps prevent misclassification of data from different classes into a single class; 2) compared to class-specific prompts, instance-specific prompts neglect the richer class-level information across multiple instances, potentially causing data from the same class to be divided into multiple classes. To effectively supplement the class-specific knowledge into existing methods, we propose a plug-and-play Class-Aware Knowledge Injection (CAKI) framework. CAKI comprises two key components, i.e., class-specific prompt generation and query-key prompt matching. The former encodes class-specific knowledge into prompts from few-shot samples that belong to the same class and stores the learned prompts in a class-level knowledge bank. The latter provides a plug-and-play mechanism for each test instance to retrieve relevant class-level knowledge from the knowledge bank and inject such knowledge to refine model predictions. Extensive experiments demonstrate that our CAKI effectively improves the performance of existing methods on base and novel classes. Code is publicly available at \href{https://github.com/yjh576/CAKI}{this https URL}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。