ProCAP：面向视觉-语言模型的概率交叉注意力提示学习

中文摘要 AI 辅助

预训练的视觉-语言模型（如CLIP）可以通过提示识别新类别，但在标注数据稀缺或测试分布发生偏移时，它们常常表现不佳。提示学习仅调整少量参数，同时保持骨干网络冻结，然而许多现有的多模态提示学习方法对视觉和文本分支的耦合较弱，在低样本场景下可能表现脆弱。我们提出ProCAP，一种概率交叉注意力提示学习框架，在不更新任何CLIP权重的情况下改善跨模态交互和训练稳定性：它学习视觉和文本提示令牌，并通过堆叠的双向多头交叉注意力将它们连接起来，使两个分支在提示深度上相互优化。为了在有限监督下减少过拟合，我们用高斯均值和方差对提示令牌进行参数化，并通过轻量级的KL散度和L2惩罚进行正则化，此外还添加了一个紧凑的对称InfoNCE头，将交叉注意的图像特征与类级文本表示在共享的低维空间中对齐。在11个数据集上的少样本基类到新类泛化、跨数据集迁移以及ImageNet偏移基准上的域泛化实验中，ProCAP在保持CLIP骨干不变的情况下，实现了强大的聚合基类到新类性能以及具有竞争力的迁移性能。

英文摘要

Pre-trained vision-language models such as CLIP can recognize new categories via prompting, but they often struggle when labeled data are scarce or the test distribution shifts. Prompt learning adapts only a small set of parameters while keeping the backbone frozen, yet many existing multimodal prompt learners couple the visual and textual branches weakly and can be brittle in low-shot regimes. We propose ProCAP, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine each other across prompt depth. To reduce overfitting under limited supervision, we parameterize prompt tokens with Gaussian means and variances and regularize them with lightweight KL and L2 penalties, and we further add a compact symmetric InfoNCE head that aligns cross-attended image features with class-level text representations in a shared low-dimensional space. Across few-shot base-to-novel generalization on 11 datasets, cross-dataset transfer, and domain generalization on ImageNet shift benchmarks, ProCAP achieves strong aggregate base-to-novel performance and competitive transfer performance while keeping the CLIP backbone unchanged.

发表机构

• University of Kurdistan（库尔德斯坦大学）
• The University of Queensland（昆士兰大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

20 pages, 8 figures, 9 tables