MaskCLIP：掩码自蒸馏推动对比语言-图像预训练
MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
AI总结：
本文提出MaskCLIP框架，将掩码自蒸馏融入对比语言-图像预训练，通过局部patch表示学习与视觉-语言对比学习互补，并在文本分支引入局部语义监督，显著提升了下游任务的linear probing、finetuning和zero-shot性能。
摘要：
本文提出了一个简单而有效的框架MaskCLIP，该框架将新提出的掩码自蒸馏融入到对比语言-图像预训练中。掩码自蒸馏的核心思想是将完整图像的表示蒸馏到从掩码图像预测出的表示中。这种融合具有两个关键优势。首先，掩码自蒸馏专注于局部patch表示学习，这与关注文本相关表示的视觉-语言对比学习形成互补。其次，从训练目标的角度来看，掩码自蒸馏与视觉-语言对比学习也保持一致，因为两者都利用视觉编码器进行特征对齐，从而能够学习局部语义并获得来自语言的间接监督。我们提供了专门设计的实验和全面分析来验证这两个优势。对称地，我们还将局部语义监督引入到文本分支中，进一步提升了预训练性能。通过大量实验，我们表明MaskCLIP在应用于各种具有挑战性的下游任务时，在语言编码器的指导下，于linear probing、finetuning和zero-shot性能上均取得了优异的结果。代码将在 https://github.com/LightDXY/MaskCLIP 发布。
英文摘要：
This paper presents a simple yet effective framework MaskCLIP, which incorporates a newly proposed masked self-distillation into contrastive language-image pretraining. The core idea of masked self-distillation is to distill representation from a full image to the representation predicted from a masked image. Such incorporation enjoys two vital benefits. First, masked self-distillation targets local patch representation learning, which is complementary to vision-language contrastive focusing on text-related representation. Second, masked self-distillation is also consistent with vision-language contrastive from the perspective of training objective as both utilize the visual encoder for feature aligning, and thus is able to learn local semantics getting indirect supervision from the language. We provide specially designed experiments with a comprehensive analysis to validate the two benefits. Symmetrically, we also introduce the local semantic supervision into the text branch, which further improves the pretraining performance. With extensive experiments, we show that MaskCLIP, when applied to various challenging downstream tasks, achieves superior results in linear probing, finetuning, and zero-shot performance with the guidance of the language encoder. Code will be release at \url{https://github.com/LightDXY/MaskCLIP}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
