解释并缓解对比多模态学习中的模态差距
Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning
AI总结：
本文通过分析对比多模态学习中的模态差距成因，提出通过温度调度和模态交换等策略缓解模态差距，从而提升多模态任务性能。
摘要：
多模态学习最近获得了显著的关注，展示了在各种零样本分类任务和一系列感知和生成应用中的出色性能。诸如对比语言-图像预训练（CLIP）之类的模型旨在通过对比学习学习共享的表示空间来弥合不同模态，如图像和文本。尽管取得了成功，但多模态学习的工作机制尚未完全理解。值得注意的是，这些模型常常表现出模态差距，即不同模态占据共享表示空间中的不同区域。在本工作中，我们深入分析了模态差距的出现，通过表征梯度流学习动态来刻画其产生。具体来说，我们识别出不匹配的数据对和一个可学习的温度参数在训练过程中导致和持续模态差距的关键作用。此外，我们的理论洞察通过在实际CLIP模型上的实验得到验证。这些发现为缓解模态差距提供了系统性的指导，包括适当的温度调度和模态交换等策略。此外，我们证明关闭模态差距可以提高图像-文本检索等任务的性能。
英文摘要：
Multimodal learning has recently gained significant popularity, demonstrating impressive performance across various zero-shot classification tasks and a range of perceptive and generative applications. Models such as Contrastive Language-Image Pretraining (CLIP) are designed to bridge different modalities, such as images and text, by learning a shared representation space through contrastive learning. Despite their success, the working mechanisms underlying multimodal learning are not yet well understood. Notably, these models often exhibit a modality gap, where different modalities occupy distinct regions within the shared representation space. In this work, we conduct an in-depth analysis of the emergence of modality gap by characterizing the gradient flow learning dynamics. Specifically, we identify the critical roles of mismatched data pairs and a learnable temperature parameter in causing and perpetuating the modality gap during training. Furthermore, our theoretical insights are validated through experiments on practical CLIP models. These findings provide principled guidance for mitigating the modality gap, including strategies such as appropriate temperature scheduling and modality swapping. Additionally, we demonstrate that closing the modality gap leads to improved performance on tasks such as image-text retrieval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
