SelfPrompt：用于鲁棒视觉语言模型自适应的置信度感知半监督微调
SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation
AI总结：
本文提出SelfPrompt，一种用于视觉语言模型的半监督提示微调方法，通过聚类引导的伪标签和置信度感知学习模块解决伪标签噪声问题，并在13个数据集上显著超越现有方法。
摘要：
我们提出了SelfPrompt，一种在半监督学习设置下用于视觉语言模型（VLMs）的新型提示微调方法。在半监督设置下微调VLMs的现有方法难以应对校准不良的VLMs对伪标签的负面影响，以及噪声伪标签的累积问题。SelfPrompt通过引入一种聚类引导的伪标签方法来提高伪标签准确率，并引入一个置信度感知的半监督学习模块，该模块通过结合监督学习和弱监督学习来最大化未标记数据的利用率，从而解决这些挑战。此外，我们还在主动半监督学习设置下研究了我们的方法，其中标记集经过战略性选择，以确保对有限标记预算的最佳利用。为此，我们提出了一种弱监督采样技术，该技术选择多样且具代表性的标记集，可无缝集成到现有方法中以增强其性能。我们在13个数据集上进行了广泛评估，使用2-shot设置，在标准半监督学习中平均提升6.23%，在主动半监督学习中平均提升6.25%，在base-to-novel泛化中平均提升4.9%，显著超越了最先进的性能。此外，SelfPrompt在单样本设置中表现出色，实现了11.78%的平均提升。
英文摘要：
We present SelfPrompt, a novel prompt-tuning approach for vision-language models (VLMs) in a semi-supervised learning setup. Existing methods for tuning VLMs in semi-supervised setups struggle with the negative impact of the miscalibrated VLMs on pseudo-labelling, and the accumulation of noisy pseudo-labels. SelfPrompt addresses these challenges by introducing a cluster-guided pseudo-labelling method that improves pseudo-label accuracy, and a confidence-aware semi-supervised learning module that maximizes the utilization of unlabelled data by combining supervised learning and weakly-supervised learning. Additionally, we investigate our method in an active semi-supervised learning setup, where the labelled set is strategically selected to ensure the best utilization of a limited labelling budget. To this end, we propose a weakly-supervised sampling technique that selects a diverse and representative labelled set, which can be seamlessly integrated into existing methods to enhance their performance. We conduct extensive evaluations across 13 datasets, significantly surpassing state-of-the-art performances with average improvements of 6.23% in standard semi-supervised learning, 6.25% in active semi-supervised learning, and 4.9% in base-to-novel generalization, using a 2-shot setup. Furthermore, SelfPrompt shows excellent generalization in single-shot settings, achieving an average improvement of 11.78%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
