为什么视觉-语言模型的提示调优对噪声标签具有鲁棒性？
Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?
AI总结：
本研究探究视觉-语言模型提示调优抗标签噪声的鲁棒性原因，发现固定类别名令牌的正则化、强预训练图文嵌入是关键，还验证了用带噪零样本预测自调优提示可提升无监督分类准确率。
摘要：
CLIP等视觉-语言模型从大规模训练数据中学习通用的文本-图像嵌入。这类模型可通过少样本提示调优适配新的分类任务。我们发现该提示调优过程对标签噪声具有极强鲁棒性，这促使我们研究促成提示调优范式鲁棒性的关键原因。我们开展了大量实验探究这一特性，发现核心因素为：1）固定的类别名令牌（classname tokens）为模型优化提供强正则化，减少噪声样本引发的梯度；2）从多样通用的网络数据中学习到的强大预训练图文嵌入，为图像分类提供了强先验知识。此外，我们证明CLIP的带噪声零样本预测可用于调优其自身提示，在无监督场景下显著提升预测准确率。代码可在https://github.com/CEWu/PTNL获取。
英文摘要：
Vision-language models such as CLIP learn a generic text-image embedding from large-scale training data. A vision-language model can be adapted to a new classification task through few-shot prompt tuning. We find that such a prompt tuning process is highly robust to label noises. This intrigues us to study the key reasons contributing to the robustness of the prompt tuning paradigm. We conducted extensive experiments to explore this property and find the key factors are: 1) the fixed classname tokens provide a strong regularization to the optimization of the model, reducing gradients induced by the noisy samples; 2) the powerful pre-trained image-text embedding that is learned from diverse and generic web data provides strong prior knowledge for image classification. Further, we demonstrate that noisy zero-shot predictions from CLIP can be used to tune its own prompt, significantly enhancing prediction accuracy in the unsupervised setting. The code is available at https://github.com/CEWu/PTNL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
