通过简单的参数高效修改进行视觉-语言模型微调
Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification
AI总结：
本文提出ClipFit方法，仅微调CLIP中特定的偏置项和归一化层，无需额外参数即可将零样本性能提升7.27%，并分析了微调对内部参数和表示的影响。
摘要：
视觉-语言模型（VLMs）微调的最新进展见证了提示微调和适配器微调的成功，而经典的对固有参数进行模型微调似乎被忽视了。人们认为，用少样本微调VLMs的参数会破坏预训练知识，因为微调CLIP模型甚至会降低性能。在本文中，我们重新审视这一观点，并提出一个新的视角：微调特定参数而非全部参数，将发掘经典模型微调在VLMs上的潜力。通过我们细致的研究，我们提出了ClipFit，这是一种简单而有效的方法，可以在不引入任何额外参数开销的情况下微调CLIP。我们证明，仅通过微调特定的偏置项和归一化层，ClipFit可以将零样本CLIP的性能平均调和均值准确率提高7.27%。最后，为了理解CLIPFit中的微调如何影响预训练模型，我们对内部参数和表示的变化进行了广泛的实验分析。我们发现，低层文本偏置层和第一层归一化层的变化远大于其他层。代码可在\url{https://github.com/minglllli/CLIPFit}获取。
英文摘要：
Recent advances in fine-tuning Vision-Language Models (VLMs) have witnessed the success of prompt tuning and adapter tuning, while the classic model fine-tuning on inherent parameters seems to be overlooked. It is believed that fine-tuning the parameters of VLMs with few-shot samples corrupts the pre-trained knowledge since fine-tuning the CLIP model even degrades performance. In this paper, we revisit this viewpoint, and propose a new perspective: fine-tuning the specific parameters instead of all will uncover the power of classic model fine-tuning on VLMs. Through our meticulous study, we propose ClipFit, a simple yet effective method to fine-tune CLIP without introducing any overhead of extra parameters. We demonstrate that by only fine-tuning the specific bias terms and normalization layers, ClipFit can improve the performance of zero-shot CLIP by 7.27\% average harmonic mean accuracy. Lastly, to understand how fine-tuning in CLIPFit affects the pre-trained models, we conducted extensive experimental analyses w.r.t. changes in internal parameters and representations. We found that low-level text bias layers and the first layer normalization layer change much more than other layers. The code is available at \url{https://github.com/minglllli/CLIPFit}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
