对比模态内与排序模态间困难负样本以增强视觉-语言组合理解
Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
AI总结：
提出一种通过对比模态内与排序模态间困难负样本的简单有效方法，改进标准图文对比学习框架以提升VLMs的组合推理能力，无需额外参数即在五个基准上取得显著提升。
摘要：
视觉-语言模型（VLMs），如CLIP，展现出强大的图文理解能力，推动了零样本图像分类、图文检索和文本生成图像等下游任务的进展。然而，现有VLMs的组合推理能力仍然欠佳。这一局限的根源在于预训练数据集中图像与字幕的对齐不足。此外，当前的对比学习目标未能关注关系、动作和属性等细粒度基础组件，导致出现“词袋”表示。我们提出了一种简单有效的方法来提升VLMs的组合推理能力。该方法通过改进和扩展标准图文对比学习框架，更好地利用了可用数据集。我们的方法不需要特定标注，也不增加额外参数。与CLIP结合后，我们的技术在五个视觉-语言组合基准上取得了优于最先进基线的显著提升。我们在 https://github.com/lezhang7/Enhance-FineGrained 开源了代码。
英文摘要：
Vision-Language Models (VLMs), such as CLIP, exhibit strong image-text comprehension abilities, facilitating advances in several downstream tasks such as zero-shot image classification, image-text retrieval, and text-to-image generation. However, the compositional reasoning abilities of existing VLMs remains subpar. The root of this limitation lies in the inadequate alignment between the images and captions in the pretraining datasets. Additionally, the current contrastive learning objective fails to focus on fine-grained grounding components like relations, actions, and attributes, resulting in "bag-of-words" representations. We introduce a simple and effective method to improve compositional reasoning in VLMs. Our method better leverages available datasets by refining and expanding the standard image-text contrastive learning framework. Our approach does not require specific annotations and does not incur extra parameters. When integrated with CLIP, our technique yields notable improvement over state-of-the-art baselines across five vision-language compositional benchmarks. We open-source our code at https://github.com/lezhang7/Enhance-FineGrained.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
