密集且对齐的字幕（DAC）促进视觉语言模型中的组合推理
Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models
AI总结：
本文针对视觉语言模型的物体偏置与组合推理缺陷，提出利用CC3M自动构建密集且图像对齐字幕的微调方法，使CLIP性能较基础模型最高提升约27%。
摘要：
视觉与语言（VL）模型为对齐图像和文本的表示空间提供了一种有效方法，催生了跨模态检索、视觉问答、字幕生成等众多应用。然而，所有主流VL模型所学习到的对齐图文空间仍受困于所谓的“物体偏置”——它们的表示表现得如同“名词袋”，大多忽略或弱化了文本/图像中描述或出现的物体属性、关系和状态。尽管近期文献中已提出一些出色尝试来解决这些“组合推理”问题，但该问题仍远未得到解决。本文揭示了限制VL模型组合推理性能的两个因素。这两个因素是用于微调和预训练VL模型的成对VL数据集的属性：（i）字幕质量，换言之即文本的“图像对齐度”；以及（ii）字幕在提及图像中出现的所有细节意义上的“密集度”。我们利用标准VL数据集（CC3M）提出了一种自动处理这些因素的微调方法。将其应用于CLIP，我们证明其组合推理性能显著提升：相比基础模型最高提升约27%，相比最强基线最高提升约20%，平均提升6.7%。
英文摘要：
Vision and Language (VL) models offer an effective method for aligning representation spaces of images and text, leading to numerous applications such as cross-modal retrieval, visual question answering, captioning, and more. However, the aligned image-text spaces learned by all the popular VL models are still suffering from the so-called `object bias' - their representations behave as `bags of nouns', mostly ignoring or downsizing the attributes, relations, and states of objects described/appearing in texts/images. Although some great attempts at fixing these `compositional reasoning' issues were proposed in the recent literature, the problem is still far from being solved. In this paper, we uncover two factors limiting the VL models' compositional reasoning performance. These two factors are properties of the paired VL dataset used for finetuning and pre-training the VL model: (i) the caption quality, or in other words `image-alignment', of the texts; and (ii) the `density' of the captions in the sense of mentioning all the details appearing on the image. We propose a fine-tuning approach for automatically treating these factors leveraging a standard VL dataset (CC3M). Applied to CLIP, we demonstrate its significant compositional reasoning performance increase of up to $\sim27\%$ over the base model, up to $\sim20\%$ over the strongest baseline, and by $6.7\%$ on average.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
