重新思考低样本CLIP适配中的视觉内容细化
Rethinking Visual Content Refinement in Low-Shot CLIP Adaptation
AI总结：
提出视觉内容细化（VCR）方法，在测试阶段通过多尺度分解与预测裕度选择合并图像视图，增强CLIP适配器对全局和局部细节的感知，在13个数据集上显著提升低样本性能。
摘要：
最近的适配方法通过有效促进知识迁移，能够提升对比视觉-语言预训练（CLIP）的低样本能力。然而，这些适配方法通常在输入图像的全局视图上操作，因此对图像的局部细节存在感知偏差。为解决这一问题，我们提出在测试阶段的适配计算之前进行视觉内容细化（VCR）。具体而言，我们首先将测试图像分解为不同尺度，以转移特征提取器对图像细节的注意力。然后，我们在每个尺度上选择具有最大预测裕度的图像视图，以过滤掉噪声图像视图，其中预测裕度由预训练的CLIP模型计算得出。最后，我们根据所选图像视图的尺度合并其内容，构建一个新的鲁棒表示。因此，合并后的内容可以直接用于帮助适配器同时关注全局和局部部分，而无需任何额外的训练参数。我们将方法应用于3个流行的低样本基准任务，涵盖13个数据集，并取得了显著优于现有最先进方法的改进。例如，在少样本分类任务中，与基线（Tip-Adapter）相比，我们的方法在无需训练和需要训练的设置下均实现了约2%的平均提升。
英文摘要：
Recent adaptations can boost the low-shot capability of Contrastive Vision-Language Pre-training (CLIP) by effectively facilitating knowledge transfer. However, these adaptation methods are usually operated on the global view of an input image, and thus biased perception of partial local details of the image. To solve this problem, we propose a Visual Content Refinement (VCR) before the adaptation calculation during the test stage. Specifically, we first decompose the test image into different scales to shift the feature extractor's attention to the details of the image. Then, we select the image view with the max prediction margin in each scale to filter out the noisy image views, where the prediction margins are calculated from the pre-trained CLIP model. Finally, we merge the content of the aforementioned selected image views based on their scales to construct a new robust representation. Thus, the merged content can be directly used to help the adapter focus on both global and local parts without any extra training parameters. We apply our method to 3 popular low-shot benchmark tasks with 13 datasets and achieve a significant improvement over state-of-the-art methods. For example, compared to the baseline (Tip-Adapter) on the few-shot classification task, our method achieves about 2\% average improvement for both training-free and training-need settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
