学会说“不”以更好地表达“是”：通过否定改进视觉-语言模型
Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations
AI总结：
本文提出CC-Neg数据集和CoN-CLIP框架，通过改进对比损失增强视觉-语言模型对否定语义的理解，在零样本分类和组合性基准上取得显著提升，同时降低了计算成本。
摘要：
现有的视觉-语言模型（VLMs）将文本描述视为一个整体单元，混淆了提示中的各个概念，从而损害了视觉语义匹配和推理能力。逻辑和语言推理中的一个重要方面是否定。本文强调了流行视觉-语言模型（如CLIP）在理解否定含义方面的局限性，即给定提示中“不”一词的作用。为了能够在包含否定的流畅提示上评估视觉-语言模型，我们提出了CC-Neg数据集，该数据集包含228,246张图像、真实描述及其对应的否定描述。通过使用CC-Neg以及对CLIP对比损失进行修改，我们提出的CoN-CLIP框架对否定有了更好的理解。这种训练范式提高了CoN-CLIP可靠编码语义的能力，在8个数据集上的零样本图像分类中，top-1准确率平均提升了3.85%。此外，CoN-CLIP在具有挑战性的组合性基准（如SugarCREPE）上比CLIP高出4.4%，展示了对文本中对象、关系和属性的涌现式组合理解能力。总体而言，我们的工作通过引入一个数据集和框架，加强了图像与文本之间的语义关联，解决了视觉-语言模型的一个关键局限性，展示了以显著降低的计算成本改进的大规模基础模型，促进了效率和可及性。
英文摘要：
Existing vision-language models (VLMs) treat text descriptions as a unit, confusing individual concepts in a prompt and impairing visual semantic matching and reasoning. An important aspect of reasoning in logic and language is negations. This paper highlights the limitations of popular VLMs such as CLIP, at understanding the implications of negations, i.e., the effect of the word "not" in a given prompt. To enable evaluation of VLMs on fluent prompts with negations, we present CC-Neg, a dataset containing 228,246 images, true captions and their corresponding negated captions. Using CC-Neg along with modifications to the contrastive loss of CLIP, our proposed CoN-CLIP framework, has an improved understanding of negations. This training paradigm improves CoN-CLIP's ability to encode semantics reliably, resulting in 3.85% average gain in top-1 accuracy for zero-shot image classification across 8 datasets. Further, CoN-CLIP outperforms CLIP on challenging compositionality benchmarks such as SugarCREPE by 4.4%, showcasing emergent compositional understanding of objects, relations, and attributes in text. Overall, our work addresses a crucial limitation of VLMs by introducing a dataset and framework that strengthens semantic associations between images and text, demonstrating improved large-scale foundation models with significantly reduced computational cost, promoting efficiency and accessibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
