重新思考少样本 CLIP 基准：归纳设置下的批判性分析
Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting
AI总结：
本文指出标准少样本 CLIP 评估因数据已被预训练见过而属于部分直推式设置，提出基于遗忘的真正归纳基准，并给出在 5880 项实验中优于 13 个基线的新方法。
摘要：
CLIP 是一个基础模型，在少样本设置中具有可迁移的分类性能。已有若干方法表明，可以利用少样本样例提升 CLIP 的性能。然而，到目前为止，所有这些技术都使用标准少样本数据集进行基准测试。我们认为，这种评估模式并不能真实反映利用少样本样例进行归纳泛化的能力。由于大多数数据集已经被 CLIP 模型见过，因此所得设置可称为部分直推式设置。为解决这一问题，我们提出一种使用遗忘技术的流程，以获得真正的归纳基线。在这一新的归纳设置中，各方法的性能显著下降（在多个数据集上的 13 个基线平均下降 55%）。我们使用 oracle 基线验证了该遗忘技术。本文还提出一种改进的少样本分类技术，在包含 5880 项实验的综合分析中，相较于其他 13 种近期基线方法持续取得最先进性能；这些实验在数据集、少样本样例数量、遗忘设置以及随机种子方面均有所变化。因此，我们指出了基于 CLIP 的少样本分类评估中存在的问题，提供了使用遗忘技术的解决方案，提出了新的基准，并给出了一种改进方法。
英文摘要：
CLIP is a foundational model with transferable classification performance in the few-shot setting. Several methods have shown improved performance of CLIP using few-shot examples. However, so far, all these techniques have been benchmarked using standard few-shot datasets. We argue that this mode of evaluation does not provide a true indication of the inductive generalization ability using few-shot examples. As most datasets have been seen by the CLIP model, the resultant setting can be termed as partially transductive. To solve this, we propose a pipeline that uses an unlearning technique to obtain true inductive baselines. In this new inductive setting, the methods show a significant drop in performance (-55% on average among 13 baselines with multiple datasets). We validate the unlearning technique using oracle baselines. An improved few-shot classification technique is proposed that consistently obtains state-of-the-art performance over 13 other recent baseline methods on a comprehensive analysis with 5880 experiments - varying the datasets, differing number of few-shot examples, unlearning setting, and with different seeds. Thus, we identify the issue with the evaluation of CLIP-based few-shot classification, provide a solution using unlearning, propose new benchmarks, and provide an improved method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
