展示并减少视觉-语言表示学习中的捷径
Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning
AI总结：
本文提出视觉-语言合成捷径框架，证明对比训练会主要学习捷径而非任务最优表示，并考察两种方法只能部分缓解该问题，揭示了对比视觉-语言表示学习中减少捷径学习的困难。
摘要：
视觉-语言模型（VLMs）主要依赖对比训练来学习图像和文本描述（captions）的通用表示。我们关注的情况是：一幅图像与多个文本描述相关联，每个文本描述既包含所有文本描述共享的信息，也包含关于图像所描绘场景的每个文本描述独有的信息。在这种情况下，尚不清楚对比损失是否足以学习包含文本描述所提供全部信息的任务最优表示，也不清楚对比学习设置是否会鼓励学习一种最小化对比损失的简单捷径。我们为视觉-语言任务引入合成捷径：一个训练和评估框架，在该框架中我们将合成捷径注入图像-文本数据。我们表明，从头开始训练或使用包含这些合成捷径的数据进行微调的对比视觉-语言模型主要学习表示该捷径的特征。因此，对比损失不足以学习任务最优表示，即包含图像与相关文本描述之间共享的所有任务相关信息的表示。我们在训练和评估框架中考察了两种减少捷径学习的方法：（i）潜在目标解码和（ii）隐式特征修改。我们通过实验表明，这两种方法都能提高评估任务上的性能，但在使用我们的捷径学习框架进行训练和评估时，只能部分减少捷径学习。因此，我们展示了我们的捷径学习框架对对比视觉-语言表示学习构成的困难和挑战。
英文摘要：
Vision-language models (VLMs) mainly rely on contrastive training to learn general-purpose representations of images and captions. We focus on the situation when one image is associated with several captions, each caption containing both information shared among all captions and unique information per caption about the scene depicted in the image. In such cases, it is unclear whether contrastive losses are sufficient for learning task-optimal representations that contain all the information provided by the captions or whether the contrastive learning setup encourages the learning of a simple shortcut that minimizes contrastive loss. We introduce synthetic shortcuts for vision-language: a training and evaluation framework where we inject synthetic shortcuts into image-text data. We show that contrastive VLMs trained from scratch or fine-tuned with data containing these synthetic shortcuts mainly learn features that represent the shortcut. Hence, contrastive losses are not sufficient to learn task-optimal representations, i.e., representations that contain all task-relevant information shared between the image and associated captions. We examine two methods to reduce shortcut learning in our training and evaluation framework: (i) latent target decoding and (ii) implicit feature modification. We show empirically that both methods improve performance on the evaluation task, but only partly reduce shortcut learning when training and evaluating with our shortcut learning framework. Hence, we show the difficulty and challenge of our shortcut learning framework for contrastive vision-language representation learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
