错位、对比再蒸馏：重新思考语言-图像预训练中的错位问题
Misalign, Contrast then Distill: Rethinking Misalignments in Language-Image Pretraining
AI总结：
针对语言-图像预训练中增强导致的图像-文本错位问题，提出错位、对比再蒸馏（MCD）度量学习方法，将错位作为额外训练信号，在多个下游数据集上达到最先进迁移性能。
摘要：
对比语言-图像预训练已成为一种利用来自网络的未整理图像-文本对来训练视觉和文本编码器的主流方法。为了提升数据效率，近期的研究引入了额外的监督项，这些监督项涉及图像的随机增强视图。然而，由于图像增强过程并不知晓其对应的文本，这一过程可能在训练期间导致不同程度的图像-文本错位。先前的方法要么忽略了这种差异，要么引入外部模型来减轻训练期间错位的影响。与此相反，我们提出了一种新颖的度量学习方法，将这种错位作为额外的训练来源加以利用，我们称之为“错位、对比再蒸馏（MCD）”。与以往将增强图像及其文本对应物视为简单正样本对的方法不同，MCD预测由增强引起的连续错位尺度。我们的大量实验结果表明，我们提出的MCD在多个分类和检索下游数据集上实现了最先进的迁移性能。
英文摘要：
Contrastive Language-Image Pretraining has emerged as a prominent approach for training vision and text encoders with uncurated image-text pairs from the web. To enhance data-efficiency, recent efforts have introduced additional supervision terms that involve random-augmented views of the image. However, since the image augmentation process is unaware of its text counterpart, this procedure could cause various degrees of image-text misalignments during training. Prior methods either disregarded this discrepancy or introduced external models to mitigate the impact of misalignments during training. In contrast, we propose a novel metric learning approach that capitalizes on these misalignments as an additional training source, which we term "Misalign, Contrast then Distill (MCD)". Unlike previous methods that treat augmented images and their text counterparts as simple positive pairs, MCD predicts the continuous scales of misalignment caused by the augmentation. Our extensive experimental results show that our proposed MCD achieves state-of-the-art transferability in multiple classification and retrieval downstream datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
