图像编码器的零样本蒸馏：如何有效利用合成数据
Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Data
AI总结：
本文研究图像编码器的零样本蒸馏，发现合成数据在对比损失下泛化性差，改用基于图像特征的L2蒸馏损失，训练出在四个数据集上性能与ViT-B/32教师模型相当且参数减少92%的学生模型。
摘要：
诸如CLIP等多模态基础模型展现出了令人印象深刻的零样本能力。然而，由于参数量大和推理时间长，其在资源受限环境中的适用性受到限制。虽然现有方法对整个CLIP架构进行了缩小，但我们专注于训练更小的图像编码器变体，这足以实现高效的零样本分类。在从较大的教师模型蒸馏表示方面，合成数据的使用已展现出前景，从而带来了强大的少样本和线性探测性能。然而，我们发现，当使用对比损失时，这种方法在真正的零样本设置中却意外地失败了。我们将对虚假特征的利用确定为导致合成数据与真实数据之间泛化性差的原因。不过，通过使用基于图像特征的L2蒸馏损失，我们缓解了这些问题并训练出学生模型，该模型在四个特定领域数据集上取得了与在DataCompXL上训练的ViT-B/32教师模型相当的零样本性能，同时参数量减少了高达92%。
英文摘要：
Multi-modal foundation models such as CLIP have showcased impressive zero-shot capabilities. However, their applicability in resource-constrained environments is limited due to their large number of parameters and high inference time. While existing approaches have scaled down the entire CLIP architecture, we focus on training smaller variants of the image encoder, which suffices for efficient zero-shot classification. The use of synthetic data has shown promise in distilling representations from larger teachers, resulting in strong few-shot and linear probe performance. However, we find that this approach surprisingly fails in true zero-shot settings when using contrastive losses. We identify the exploitation of spurious features as being responsible for poor generalization between synthetic and real data. However, by using the image feature-based L2 distillation loss, we mitigate these problems and train students that achieve zero-shot performance which on four domain-specific datasets is on-par with a ViT-B/32 teacher model trained on DataCompXL, while featuring up to 92% fewer parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
