揭示CLIP中的骨干网络效应：探索表征协同与差异
Unveiling Backbone Effects in CLIP: Exploring Representational Synergies and Variances
AI总结：
本研究揭示不同骨干架构在CLIP中的表征差异，提出组合预测方法，性能提升达6.34%，选择适当骨干可提升超20%。
摘要：
对比语言-图像预训练（CLIP）作为一种突出的图像表征学习方法脱颖而出。各种神经架构，从基于Transformer的模型（如视觉Transformer（ViT））到卷积网络（如ResNet），均使用CLIP进行训练，并作为多种视觉任务中的通用骨干网络。尽管使用相同的数据和训练目标，这些架构学习到的表征的有效性引发了一个关键问题。我们的研究探讨了这些骨干架构在CLIP性能上的差异，揭示了它们在分类上的显著差异。值得注意的是，对这些表征进行归一化会导致性能的显著变化。我们的发现展示了骨干网络预测之间一种显著的潜在协同作用，通过明智地选择适当的骨干网络，性能可提升超过20%。此外，我们提出了一种简单而有效的方法来组合多个骨干网络的预测，带来了高达6.34%的显著性能提升。我们将发布用于复现结果的代码。
英文摘要：
Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various neural architectures, spanning Transformer-based models like Vision Transformers (ViTs) to Convolutional Networks (ConvNets) like ResNets, are trained with CLIP and serve as universal backbones across diverse vision tasks. Despite utilizing the same data and training objectives, the effectiveness of representations learned by these architectures raises a critical question. Our investigation explores the differences in CLIP performance among these backbone architectures, revealing significant disparities in their classifications. Notably, normalizing these representations results in substantial performance variations. Our findings showcase a remarkable possible synergy between backbone predictions that could reach an improvement of over 20% through informed selection of the appropriate backbone. Moreover, we propose a simple, yet effective approach to combine predictions from multiple backbones, leading to a notable performance boost of up to 6.34\%. We will release the code for reproducing the results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
