arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.13808cs.CV
收藏

CoAPT:用于提示调优的上下文属性词

CoAPT: Context Attribute words for Prompt Tuning

Kookmin University(国民大学);Hanyang University(汉阳大学)

Gun Lee, Subin An, Sungyong Baik, Soochahn Lee

AI总结:

CoAPT通过将属性词作为附加提示集成到可学习提示调优中,并训练元网络生成特征偏置,以改善CLIP文本与图像嵌入的对齐,显著提升少样本/零样本图像分类性能。

摘要:

我们提出了一种新颖的提示调优方法,称为CoAPT(上下文属性词提示调优),用于少样本/零样本图像分类。核心动机是属性是描述性词语,包含关于给定概念的丰富信息。因此,我们旨在丰富现有提示调优方法的文本查询,改善CLIP嵌入空间中文本与图像嵌入之间的对齐。为此,CoAPT将属性词作为附加提示集成到可学习的提示调优中,并且可以轻松地纳入各种现有的提示调优方法。为了促进属性融入文本嵌入以及与图像嵌入的对齐,软提示与一个额外的元网络一起训练,该元网络从图像-文本组合查询的拼接特征编码中生成输入图像特定的特征偏置。我们的实验表明,CoAPT在多个少样本/零样本图像分类任务上显著提升了现有基线方法的性能,包括基础到新颖类泛化、跨数据集迁移和域泛化。我们的发现强调了硬提示与软提示结合的重要性,并为未来关于预训练模型中文本与图像潜在空间相互作用的研究铺平了道路。

英文摘要:

We propose a novel prompt tuning method called CoAPT(Context Attribute words in Prompt Tuning) for few/zero-shot image classification. The core motivation is that attributes are descriptive words with rich information about a given concept. Thus, we aim to enrich text queries of existing prompt tuning methods, improving alignment between text and image embeddings in CLIP embedding space. To do so, CoAPT integrates attribute words as additional prompts within learnable prompt tuning and can be easily incorporated into various existing prompt tuning methods. To facilitate the incorporation of attributes into text embeddings and the alignment with image embeddings, soft prompts are trained together with an additional meta-network that generates input-image-wise feature biases from the concatenated feature encodings of the image-text combined queries. Our experiments demonstrate that CoAPT leads to considerable improvements for existing baseline methods on several few/zero-shot image classification tasks, including base-to-novel generalization, cross-dataset transfer, and domain generalization. Our findings highlight the importance of combining hard and soft prompts and pave the way for future research on the interplay between text and image latent spaces in pre-trained models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑