arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.04971cs.LG
收藏

理解多模态对比学习对分布偏移的鲁棒性

Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift

University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

Yihao Xue, Siddharth Joshi, Dang Nguyen, Baharan Mirzasoleiman

AI总结:

本研究揭示了多模态对比学习(如CLIP)对分布偏移鲁棒性的两种机制:类内对比与类间特征共享,并通过理论分析与实验验证了其提升零样本分类准确率的效果。

摘要:

近年来,多模态对比学习(MMCL)方法,如CLIP,在学习对分布偏移具有鲁棒性并能泛化到新领域的表示方面取得了显著成功。尽管取得了经验上的成功,但学习这种可泛化表示的机制尚不明确。在这项工作中,我们严格分析了这一问题,并揭示了MMCL鲁棒性背后的两种机制:\u201c类内对比\u201d,它允许模型学习具有高方差的特征;以及\u201c类间特征共享\u201d,其中一个类中的标注细节有助于更好地学习其他类。这两种机制都防止了训练数据中过度表示的虚假特征掩盖可泛化的核心特征。这导致了在分布偏移下优越的零样本分类准确率。此外,我们从理论上证明了使用丰富标题对鲁棒性的益处,并探讨了在标题中标注不同类型细节的效果。我们通过实验验证了我们的理论发现,包括一个精心设计的合成实验和一个在MSCOCO/Conceptual Captions上训练CLIP模型并在偏移的ImageNets上评估它们的实验。

英文摘要:

Recently, multimodal contrastive learning (MMCL) approaches, such as CLIP, have achieved a remarkable success in learning representations that are robust against distribution shift and generalize to new domains. Despite the empirical success, the mechanism behind learning such generalizable representations is not understood. In this work, we rigorously analyze this problem and uncover two mechanisms behind MMCL's robustness: \emph{intra-class contrasting}, which allows the model to learn features with a high variance, and \emph{inter-class feature sharing}, where annotated details in one class help learning other classes better. Both mechanisms prevent spurious features that are over-represented in the training data to overshadow the generalizable core features. This yields superior zero-shot classification accuracy under distribution shift. Furthermore, we theoretically demonstrate the benefits of using rich captions on robustness and explore the effect of annotating different types of details in the captions. We validate our theoretical findings through experiments, including a well-designed synthetic experiment and an experiment involving training CLIP models on MSCOCO/Conceptual Captions and evaluating them on shifted ImageNets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑