Nemesis：归一化视觉-语言模型的软提示向量
Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models
AI总结：
本文揭示了VLMs软提示向量的低范数效应，并提出Nemesis方法对其进行归一化，首次系统研究了软提示向量范数对模型性能的影响，为软提示微调提供了新见解。
摘要：
随着CLIP等大规模预训练视觉-语言模型（VLMs）的普及，软提示微调已成为将这些模型适配到各种下游任务的流行方法。然而，很少有工作深入研究可学习软提示向量的内在属性，特别是其范数对VLMs性能的影响。这促使我们提出一个未被探索的研究问题：“我们是否需要在VLMs中归一化软提示？”为填补这一研究空白，我们首先通过进行广泛的破坏实验揭示了一种现象，称为低范数效应，表明降低某些已学习提示的范数偶尔会提升VLMs的性能，而增加范数通常会降低其性能。为了利用这一效应，我们提出了一种名为Nemesis（归一化视觉-语言模型的软提示向量）的新方法，用于在VLMs中归一化软提示向量。据我们所知，我们的工作是首个系统研究VLMs中软提示向量范数作用的工作，为未来的软提示微调研究提供了有价值的见解。代码可在https://github.com/ShyFoo/Nemesis获取。
英文摘要：
With the prevalence of large-scale pretrained vision-language models (VLMs), such as CLIP, soft-prompt tuning has become a popular method for adapting these models to various downstream tasks. However, few works delve into the inherent properties of learnable soft-prompt vectors, specifically the impact of their norms to the performance of VLMs. This motivates us to pose an unexplored research question: ``Do we need to normalize the soft prompts in VLMs?'' To fill this research gap, we first uncover a phenomenon, called the \textbf{Low-Norm Effect} by performing extensive corruption experiments, suggesting that reducing the norms of certain learned prompts occasionally enhances the performance of VLMs, while increasing them often degrades it. To harness this effect, we propose a novel method named \textbf{N}ormalizing th\textbf{e} soft-pro\textbf{m}pt v\textbf{e}ctors of vi\textbf{si}on-language model\textbf{s} (\textbf{Nemesis}) to normalize soft-prompt vectors in VLMs. To the best of our knowledge, our work is the first to systematically investigate the role of norms of soft-prompt vector in VLMs, offering valuable insights for future research in soft-prompt tuning. The code is available at \texttt{\href{https://github.com/ShyFoo/Nemesis}{https://github.com/ShyFoo/Nemesis}}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
