重新思考视觉-语言模型的两阶段少样本适配
Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages
AI总结：
本文提出两阶段少样本适配方法 2SFS，将固定计算预算分别用于 PEFT 学习任务特征提取器和训练线性分类器，并支持类别级选择性推理。
摘要：
训练分类器的一种传统方案是：(i) 学习一个良好的特征提取器，以及 (ii) 在其上优化一个线性层。当每个类别只有少量样本可用时，如少样本适配（Few-Shot Adaptation, FSA）场景，数据不足以拟合大量参数，从而使上述方案变得不切实际。对于大型预训练视觉-语言模型（Vision-Language Models, VLMs）而言尤其如此，这推动了参数高效微调（Parameter-Efficient Fine-tuning, PEFT）与 FSA 交叉领域的成功研究。在本文中，我们首先分析 PEFT 技术在仅使用类别子集（称为“基类”）的少样本数据进行训练时的学习动态。我们表明，这种动态会自然分裂为两个不同阶段：(i) 任务级特征提取，以及 (ii) 对可用概念的专门化。为适应这种动态，我们随后脱离基于提示或适配器的方法，以不同方式处理 FSA。具体而言，在固定计算预算下，我们将其划分为：(i) 通过 PEFT 学习任务特定的特征提取器，以及 (ii) 在其上训练一个线性分类器。我们将该方案称为两阶段少样本适配（Two-Stage Few-Shot Adaptation, 2SFS）。与既有方法不同，我们的方案实现了一种新的类别级选择性推理：在测试时，只有新类别由适配后的文本编码器生成嵌入，而基类的嵌入已存在于分类器中。在两种设置、三种骨干网络和十一个数据集上使用固定超参数的结果表明，2SFS 达到或超过最先进水平，而既有方法在不同设置间性能显著下降。
英文摘要：
An old-school recipe for training a classifier is to (i) learn a good feature extractor and (ii) optimize a linear layer atop. When only a handful of samples are available per category, as in Few-Shot Adaptation (FSA), data are insufficient to fit a large number of parameters, rendering the above impractical. This is especially true with large pre-trained Vision-Language Models (VLMs), which motivated successful research at the intersection of Parameter-Efficient Fine-tuning (PEFT) and FSA. In this work, we start by analyzing the learning dynamics of PEFT techniques when trained on few-shot data from only a subset of categories, referred to as the ``base'' classes. We show that such dynamics naturally splits into two distinct phases: (i) task-level feature extraction and (ii) specialization to the available concepts. To accommodate this dynamic, we then depart from prompt- or adapter-based methods and tackle FSA differently. Specifically, given a fixed computational budget, we split it to (i) learn a task-specific feature extractor via PEFT and (ii) train a linear classifier on top. We call this scheme Two-Stage Few-Shot Adaptation (2SFS). Differently from established methods, our scheme enables a novel form of selective inference at a category level, i.e., at test time, only novel categories are embedded by the adapted text encoder, while embeddings of base categories are available within the classifier. Results with fixed hyperparameters across two settings, three backbones, and eleven datasets, show that 2SFS matches or surpasses the state-of-the-art, while established methods degrade significantly across settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
