完全微调的CLIP模型是高效的少样本学习器
Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners
AI总结：
本文提出CLIP-CITE框架，通过判别式视觉-文本任务和知识蒸馏对完整CLIP模型进行细致微调，在少样本及多种泛化场景下提升特定任务性能并保持模型通用性。
摘要：
提示调优通过训练一小部分参数，能够有效地将预训练的视觉-语言模型（VLMs）适配到下游任务。然而，当调优后的模型被应用于不同数据集或领域时，这种方法往往以牺牲灵活性和适应性为代价。在本文中，我们探索通过对整个VLM进行细致优化，以最小的参数调整来捕获任务特定信息。在有限监督下针对特定任务微调整个VLM时，过拟合和灾难性遗忘成为实际面临的主要因素。为了缓解这些问题，我们提出了一个名为CLIP-CITE的框架，通过设计一种判别式视觉-文本任务，以监督方式进一步对齐视觉-文本语义，并整合知识蒸馏技术来保留已获得的知识。在少样本学习、基类到新类泛化、领域泛化和跨领域泛化设置下的大量实验结果表明，我们的方法在有限监督下有效提升了特定任务的性能，同时保持了VLM在其他数据集上的通用性。
英文摘要：
Prompt tuning, which involves training a small set of parameters, effectively enhances the pre-trained Vision-Language Models (VLMs) to downstream tasks. However, they often come at the cost of flexibility and adaptability when the tuned models are applied to different datasets or domains. In this paper, we explore capturing the task-specific information via meticulous refinement of entire VLMs, with minimal parameter adjustments. When fine-tuning the entire VLMs for specific tasks under limited supervision, overfitting and catastrophic forgetting become the defacto factors. To mitigate these issues, we propose a framework named CLIP-CITE via designing a discriminative visual-text task, further aligning the visual-text semantics in a supervision manner, and integrating knowledge distillation techniques to preserve the gained knowledge. Extensive experimental results under few-shot learning, base-to-new generalization, domain generalization, and cross-domain generalization settings, demonstrate that our method effectively enhances the performance on specific tasks under limited supervision while preserving the versatility of the VLMs on other datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
