使基于VLM微调的鲁棒少样本识别得以验证
Enabling Validation for Robust Few-Shot Recognition
AI总结：
本文提出VEST框架，通过验证启用的分阶段微调策略，解决少样本识别中验证数据不足的问题，提升模型在ID和OOD数据上的泛化能力。
摘要：
少样本识别（FSR）通过使用极少量的任务特定标记数据进行训练来解决分类任务。现有方法通过适配或微调预训练的视觉-语言模型（VLM）并从开放数据源检索相关但嘈杂的样本来增强稀缺的训练数据。微调后的VLM在任务特定的内部分布（ID）测试数据上表现良好，但在外部分布（OOD）测试数据上表现不佳。这促使我们研究基于VLM微调的鲁棒FSR。FSR的核心挑战是数据稀缺，这不仅限于有限的训练数据，还扩展到完全缺乏验证数据。我们识别出一个关键悖论作为潜在解决方案：将检索到的开放数据用于验证。由于这些检索到的数据本质上与任务特定的内部分布训练数据相比是外部分布的，微调后的VLM在这些数据上表现下降。这导致验证逻辑倾向于预训练模型而无需微调，阻碍了泛化能力的改进。为了解决这一困境，我们引入了一种新的验证策略，该策略在少量样本ID数据和检索数据上分别实现性能提升和降级。我们的验证使参数选择用于部分微调和检查点选择，缓解了过拟合并提高了测试数据的泛化能力。我们将这一策略与鲁棒学习统一成一个整体框架：验证启用的分阶段微调（VEST）。在已建立的ImageNet OOD基准上进行的广泛实验表明，VEST显著优于现有VLM适应方法，在ID和OOD数据上均实现了最先进的FSR性能。
英文摘要：
Few-Shot Recognition (FSR) tackles classification tasks by training with minimal task-specific labeled data. Prevailing methods adapt or finetune a pretrained Vision-Language Model (VLM) and augment the scarce training data by retrieving task-relevant but noisy samples from open data sources. The finetuned VLM generalizes decently well to the task-specific in-distribution (ID) test data but struggles with out-of-distribution (OOD) test data. This motivates our study of robust FSR with VLM finetuning. The core challenge of FSR is data scarcity, extending beyond limited training data to a complete lack of validation data. We identify a key paradox as a potential solution: repurposing the retrieved open data for validation. As such retrieved data are inherently OOD compared with the task-specific ID training data, finetuned VLMs yield degraded performance on the retrieved data. This causes the validation logic to favor the pretrained model without any finetuning, hindering improvements w.r.t generalization. To resolve this dilemma, we introduce a novel validation strategy that harmonizes performance gain and degradation on the few-shot ID data and the retrieved data, respectively. Our validation enables parameter selection for partial finetuning and checkpoint selection, mitigating overfitting and improving test-data generalization. We unify this strategy with robust learning into a cohesive framework: Validation-Enabled Stage-wise Tuning (VEST). Extensive experiments on the established ImageNet OOD benchmarks show that VEST significantly outperforms existing VLM adaptation methods, achieving state-of-the-art FSR performance on both ID and OOD data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
