CLIP-Adapter：通过特征适配器实现更好的视觉-语言模型
CLIP-Adapter: Better Vision-Language Models with Feature Adapters
AI总结：
本文提出CLIP-Adapter，通过特征适配器在视觉或语言分支进行微调，采用瓶颈层学习新特征并与预训练特征残差融合，超越提示调优，在视觉分类任务上验证有效性。
摘要：
大规模对比视觉-语言预训练在视觉表示学习方面已显示出显著进展。与通过固定离散标签集训练的传统视觉系统不同，文献[radford2021learning]引入了一种新范式，直接在开放词汇环境中学习将图像与原始文本对齐。在下游任务中，采用精心选择的文本提示进行零样本预测。为避免非平凡的提示工程，文献[zhou2021coop]提出了上下文优化，利用少量训练样本学习连续向量作为任务特定提示。本文表明，除了提示调优之外，还有另一条实现更好视觉-语言模型的路径。提示调优针对文本输入，而我们提出CLIP-Adapter，在视觉或语言分支上使用特征适配器进行微调。具体而言，CLIP-Adapter采用额外的瓶颈层来学习新特征，并与原始预训练特征进行残差式特征融合。因此，CLIP-Adapter能够超越上下文优化，同时保持设计简单。在各种视觉分类任务上的实验和大量消融研究证明了我们方法的有效性。代码已发布在https://github.com/gaopengcuhk/CLIP-Adapter。
英文摘要：
Large-scale contrastive vision-language pre-training has shown significant progress in visual representation learning. Unlike traditional visual systems trained by a fixed set of discrete labels, a new paradigm was introduced in \cite{radford2021learning} to directly learn to align images with raw texts in an open-vocabulary setting. On downstream tasks, a carefully chosen text prompt is employed to make zero-shot predictions.~To avoid non-trivial prompt engineering, context optimization \cite{zhou2021coop} has been proposed to learn continuous vectors as task-specific prompts with few-shot training examples.~In this paper, we show that there is an alternative path to achieve better vision-language models other than prompt tuning.~While prompt tuning is for the textual inputs, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Specifically, CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending with the original pre-trained features.~As a consequence, CLIP-Adapter is able to outperform context optimization while maintains a simple design. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach. Code is released at t https://github.com/gaopengcuhk/CLIP-Adapter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
