CapS-Adapter：基于标题的多模态适配器在零样本分类中的应用
CapS-Adapter: Caption-based MultiModal Adapter in Zero-Shot Classification
AI总结：
本文提出CapS-Adapter，一种基于标题支持集的免训练多模态适配器，利用CLIP的多模态特征提升零样本分类性能，在19个数据集上平均准确率提升2.19%。
摘要：
近期视觉-语言基础模型（如CLIP）的进展在零样本分类中取得了显著成果。然而，像CLIP这样的模型参数规模庞大，导致微调过程资源密集。为此，TIP-Adapter和SuS-X引入了免训练方法，旨在提升下游任务的效能。这些方法虽采用支持集以保持知识缓存与测试集之间的数据分布一致性，但在面对测试数据分布差异显著时，往往在测试集上的泛化能力不足。在本工作中，我们提出CapS-Adapter，一种创新方法，采用基于标题的支持集，有效利用图像和标题特征，在免训练场景中超越现有最先进技术。CapS-Adapter巧妙构建紧密反映目标分布的支持集，利用从多模态大模型中提取的实例级分布特征。通过利用CLIP的单模态和跨模态优势，CapS-Adapter通过多模态支持集提升预测准确性。我们的方法在19个基准数据集上取得了卓越的零样本分类结果，准确率较先前领先方法提升2.19%。我们的贡献通过在多个基准数据集上的广泛验证得到证实，展示了优越的性能和强大的泛化能力。我们的代码已在https://github.com/WLuLi/CapS-Adapter公开提供。
英文摘要：
Recent advances in vision-language foundational models, such as CLIP, have demonstrated significant strides in zero-shot classification. However, the extensive parameterization of models like CLIP necessitates a resource-intensive fine-tuning process. In response, TIP-Adapter and SuS-X have introduced training-free methods aimed at bolstering the efficacy of downstream tasks. While these approaches incorporate support sets to maintain data distribution consistency between knowledge cache and test sets, they often fall short in terms of generalization on the test set, particularly when faced with test data exhibiting substantial distributional variations. In this work, we present CapS-Adapter, an innovative method that employs a caption-based support set, effectively harnessing both image and caption features to exceed existing state-of-the-art techniques in training-free scenarios. CapS-Adapter adeptly constructs support sets that closely mirror target distributions, utilizing instance-level distribution features extracted from multimodal large models. By leveraging CLIP's single and cross-modal strengths, CapS-Adapter enhances predictive accuracy through the use of multimodal support sets. Our method achieves outstanding zero-shot classification results across 19 benchmark datasets, improving accuracy by 2.19\% over the previous leading method. Our contributions are substantiated through extensive validation on multiple benchmark datasets, demonstrating superior performance and robust generalization capabilities. Our code is made publicly available at https://github.com/WLuLi/CapS-Adapter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
