重新审视视觉-语言模型的提示预训练
Revisiting Prompt Pretraining of Vision-Language Models
AI总结：
针对大规模提示预训练中可学习提示欠拟合且泛化差的问题，本文提出RPP框架，通过非共享Q/K/V提示结构和CLIP教师软标签监督，提升拟合与跨任务迁移性能。
摘要：
提示学习是定制视觉-语言模型（VLM）以适应各种下游任务的有效方法，其过程涉及调优输入提示标记中极少的参数。近来，在大规模数据集（例如 ImageNet-21K）上进行的提示预训练，已在面向通用视觉判别的提示学习中发挥关键作用。然而，我们通过重新审视观察到，在提示预训练期间，面对海量图像，数量有限的可学习提示可能面临欠拟合风险，同时导致泛化性能较差。为解决上述问题，本文提出一个名为 Revisiting Prompt Pretraining（RPP）的通用框架，旨在从提示结构和提示监督两个方面提升拟合与泛化能力。在提示结构方面，我们打破了通常做法中 query、key 和 value 向量均派生自共享可学习提示标记的限制；相反，我们引入非共享的独立 query、key 和 value 可学习提示，从而通过增加参数多样性增强模型的拟合能力。在提示监督方面，我们额外使用由预训练的 Contrastive Language Image Pretraining（CLIP）教师模型提供的零样本概率预测所导出的软标签。这些软标签对类间关系产生更细致、更通用的洞察，从而赋予预训练过程更好的泛化能力。RPP 生成更具韧性的提示初始化，增强其在多样化视觉识别任务上的稳健可迁移性。在多个基准上的实验一致证实，我们预训练的提示达到最先进（SOTA）性能。代码和模型将很快公开。
英文摘要：
Prompt learning is an effective method to customize Vision-Language Models (VLMs) for various downstream tasks, involving tuning very few parameters of input prompt tokens. Recently, prompt pretraining in large-scale dataset (e.g., ImageNet-21K) has played a crucial role in prompt learning for universal visual discrimination. However, we revisit and observe that the limited learnable prompts could face underfitting risks given the extensive images during prompt pretraining, simultaneously leading to poor generalization. To address the above issues, in this paper, we propose a general framework termed Revisiting Prompt Pretraining (RPP), which targets at improving the fitting and generalization ability from two aspects: prompt structure and prompt supervision. For prompt structure, we break the restriction in common practice where query, key, and value vectors are derived from the shared learnable prompt token. Instead, we introduce unshared individual query, key, and value learnable prompts, thereby enhancing the model's fitting capacity through increased parameter diversity. For prompt supervision, we additionally utilize soft labels derived from zero-shot probability predictions provided by a pretrained Contrastive Language Image Pretraining (CLIP) teacher model. These soft labels yield more nuanced and general insights into the inter-class relationships, thereby endowing the pretraining process with better generalization ability. RPP produces a more resilient prompt initialization, enhancing its robust transferability across diverse visual recognition tasks. Experiments across various benchmarks consistently confirm the state-of-the-art (SOTA) performance of our pretrained prompts. Codes and models will be made available soon.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
