COSMOS：用于视觉语言预训练的跨模态自蒸馏
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
AI总结：
提出COSMOS，一种结合文本裁剪策略与交叉注意力模块的跨模态自蒸馏框架，通过多模态增强学习全面表示，在零样本检索、分类和分割等任务上超越强基线。
摘要：
基于对比损失训练的视觉语言模型（VLMs）在各种视觉和语言任务中取得了显著进展。然而，对比损失的全局性使得VLMs主要关注前景物体，忽略了图像中的其他关键信息，这限制了它们在下游任务中的有效性。为了解决这些挑战，我们提出了COSMOS：用于视觉语言预训练的跨模态自蒸馏方法，该方法将一种新颖的文本裁剪策略和交叉注意力模块整合到一个自监督学习框架中。我们创建了图像和文本的全局与局部视图（即多模态增强），这对于VLMs中的自蒸馏至关重要。我们进一步引入了交叉注意力模块，使COSMOS能够学习全面的跨模态表示，并通过跨模态自蒸馏损失进行优化。COSMOS在各种零样本下游任务（包括检索、分类和语义分割）上持续优于先前的强基线。此外，它在视觉感知和上下文理解任务上超越了在更大数据集上训练的基于CLIP的模型。代码可在https://github.com/ExplainableML/cosmos获取。
英文摘要：
Vision-Language Models (VLMs) trained with contrastive loss have achieved significant advancements in various vision and language tasks. However, the global nature of the contrastive loss makes VLMs focus predominantly on foreground objects, neglecting other crucial information in the image, which limits their effectiveness in downstream tasks. To address these challenges, we propose COSMOS: CrOSs-MOdality Self-distillation for vision-language pre-training that integrates a novel text-cropping strategy and cross-attention module into a self-supervised learning framework. We create global and local views of images and texts (i.e., multi-modal augmentations), which are essential for self-distillation in VLMs. We further introduce a cross-attention module, enabling COSMOS to learn comprehensive cross-modal representations optimized via a cross-modality self-distillation loss. COSMOS consistently outperforms previous strong baselines on various zero-shot downstream tasks, including retrieval, classification, and semantic segmentation. Additionally, it surpasses CLIP-based models trained on larger datasets in visual perception and contextual understanding tasks. Code is available at https://github.com/ExplainableML/cosmos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
