ProKeR：大规模视觉-语言模型少样本适应的核视角
ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models
AI总结：
从核视角分析Tip-Adapter，提出全局近端正则化的ProKeR方法，在11个数据集上达到少样本适应最先进性能。
摘要：
对比语言-图像预训练（CLIP）的日益流行使其在各种视觉下游任务中得到广泛应用。为了提升CLIP的有效性和多功能性，高效的少样本适应技术已被广泛采用。在这些方法中，免训练方法，特别是以Tip-Adapter为代表的缓存方法，因其无需额外微调即可实现轻量级适应而受到关注。本文从核视角重新审视Tip-Adapter，表明缓存方法作为局部适配器，并与成熟的核文献相关联。基于这一见解，我们提供了对这些方法运作方式的理论理解，并提出了多种增强Tip-Adapter基线的途径。值得注意的是，我们的分析显示了在局部适配器中纳入全局信息的重要性。因此，我们随后提出了一种全局方法，该方法在再生核希尔伯特空间（RKHS）中利用CLIP作为基学习器学习近端正则化器。我们的方法称为ProKeR（近端核岭回归），具有闭式解，并在标准少样本适应基准的11个数据集上取得了最先进的性能。
英文摘要：
The growing popularity of Contrastive Language-Image Pretraining (CLIP) has led to its widespread application in various visual downstream tasks. To enhance CLIP's effectiveness and versatility, efficient few-shot adaptation techniques have been widely adopted. Among these approaches, training-free methods, particularly caching methods exemplified by Tip-Adapter, have gained attention for their lightweight adaptation without the need for additional fine-tuning. In this paper, we revisit Tip-Adapter from a kernel perspective, showing that caching methods function as local adapters and are connected to a well-established kernel literature. Drawing on this insight, we offer a theoretical understanding of how these methods operate and suggest multiple avenues for enhancing the Tip-Adapter baseline. Notably, our analysis shows the importance of incorporating global information in local adapters. Therefore, we subsequently propose a global method that learns a proximal regularizer in a reproducing kernel Hilbert space (RKHS) using CLIP as a base learner. Our method, which we call ProKeR (Proximal Kernel ridge Regression), has a closed form solution and achieves state-of-the-art performances across 11 datasets in the standard few-shot adaptation benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
