CLIP 中的协同性与多样性：通过自适应骨干网络集成提升性能
Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling
AI总结：
本文分析不同 CLIP 视觉骨干网络的表征、性能与鲁棒性差异，提出每类仅需一个标签样本的自适应骨干集成方法，最高较最佳单模型提升 39.1% 准确率。
摘要：
Contrastive Language-Image Pretraining（CLIP，对比语言-图像预训练）是图像表征学习中一种引人注目的重要方法。从 vision transformers（ViT）到卷积网络（ResNets），多种架构都已使用 CLIP 进行训练，以作为应对多样化视觉任务的通用解决方案。本文探究了各种经 CLIP 训练的视觉骨干网络之间的差异。尽管使用相同的数据和训练目标，我们发现这些架构具有显著不同的表征、在不同数据集上具有不同的分类性能，并且对某些类型的图像扰动具有不同的鲁棒性属性。我们的发现表明，通过利用各骨干网络各自的优势，它们之间存在显著的潜在协同作用。原则上，如果能够依据每个测试样本有依据地选择最优骨干网络，分类准确率可以提高超过 40 个百分点。基于这一洞见，我们开发了一种简单而强大的方法来自适应地集成多个骨干网络。该方法每类仅使用少至一个带标签样本来调节骨干网络的自适应组合。在大量数据集集合上，该方法相较于最佳单一骨干网络实现了高达 39.1% 的显著准确率提升，远远超过传统集成方法。
英文摘要：
Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various architectures, from vision transformers (ViTs) to convolutional networks (ResNets) have been trained with CLIP to serve as general solutions to diverse vision tasks. This paper explores the differences across various CLIP-trained vision backbones. Despite using the same data and training objective, we find that these architectures have notably different representations, different classification performance across datasets, and different robustness properties to certain types of image perturbations. Our findings indicate a remarkable possible synergy across backbones by leveraging their respective strengths. In principle, classification accuracy could be improved by over 40 percentage with an informed selection of the optimal backbone per test example.Using this insight, we develop a straightforward yet powerful approach to adaptively ensemble multiple backbones. The approach uses as few as one labeled example per class to tune the adaptive combination of backbones. On a large collection of datasets, the method achieves a remarkable increase in accuracy of up to 39.1% over the best single backbone, well beyond traditional ensembles
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
