Zoom-shot：利用多模态损失将 CLIP 快速高效地无监督零样本迁移到视觉编码器
Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss
AI总结：
Zoom-shot 提出利用循环一致性损失和提示引导知识蒸馏损失，在单个 epoch 内以无监督非配对数据将 CLIP 零样本能力迁移到任意预训练视觉编码器，并在粗、细粒度分类上超越此前最优结果。
摘要：
视觉与语言的融合通过视觉-语言模型（VLMs）的出现为计算机视觉带来了变革性转变。然而，现有 VLMs 的资源密集型特性构成了重大挑战。我们需要一种可及的方法来开发下一代 VLMs。为解决这一问题，我们提出 Zoom-shot，这是一种将 CLIP 的零样本能力迁移到任意预训练视觉编码器的新方法。我们通过使用专门设计的多模态损失函数，利用 CLIP 潜在空间中存在的多模态信息（即文本和图像）来实现这一点。这些损失函数是（1）循环一致性损失和（2）我们提出的提示引导知识蒸馏损失（PG-KD）。PG-KD 将知识蒸馏的概念与 CLIP 的零样本分类相结合，以捕捉文本和图像特征之间的交互。借助我们的多模态损失，我们仅在单个 epoch 内训练 CLIP 潜在空间与预训练视觉编码器潜在空间之间的线性映射。此外，Zoom-shot 完全无监督，并使用非配对数据进行训练。我们在粗粒度和细粒度分类数据集上测试了一系列被增强为新 VLMs 的视觉编码器的零样本能力，超越了该问题领域此前的 state-of-the-art。在消融研究中，我们发现 Zoom-shot 允许在训练期间在数据和计算之间进行权衡；并且我们的 state-of-the-art 结果可以通过将训练从 ImageNet 训练数据的 20% 减少到 1% 并训练 20 个 epoch 来获得。所有代码和模型均可在 GitHub 上获取。
英文摘要：
The fusion of vision and language has brought about a transformative shift in computer vision through the emergence of Vision-Language Models (VLMs). However, the resource-intensive nature of existing VLMs poses a significant challenge. We need an accessible method for developing the next generation of VLMs. To address this issue, we propose Zoom-shot, a novel method for transferring the zero-shot capabilities of CLIP to any pre-trained vision encoder. We do this by exploiting the multimodal information (i.e. text and image) present in the CLIP latent space through the use of specifically designed multimodal loss functions. These loss functions are (1) cycle-consistency loss and (2) our novel prompt-guided knowledge distillation loss (PG-KD). PG-KD combines the concept of knowledge distillation with CLIP's zero-shot classification, to capture the interactions between text and image features. With our multimodal losses, we train a $\textbf{linear mapping}$ between the CLIP latent space and the latent space of a pre-trained vision encoder, for only a $\textbf{single epoch}$. Furthermore, Zoom-shot is entirely unsupervised and is trained using $\textbf{unpaired}$ data. We test the zero-shot capabilities of a range of vision encoders augmented as new VLMs, on coarse and fine-grained classification datasets, outperforming the previous state-of-the-art in this problem domain. In our ablations, we find Zoom-shot allows for a trade-off between data and compute during training; and our state-of-the-art results can be obtained by reducing training from 20% to 1% of the ImageNet training data with 20 epochs. All code and models are available on GitHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
