自然语言推理提升视觉-语言模型的组合性
Natural Language Inference Improves Compositionality in Vision-Language Models
AI总结：
针对视觉-语言模型组合推理能力弱的问题，提出利用自然语言推理生成多样蕴涵/矛盾句的CECE方法，无需额外微调即可在多个基准上取得显著性能提升。
摘要：
视觉-语言模型（Vision-Language Models, VLMs）的组合推理仍然具有挑战性，因为这类模型通常难以关联物体、属性和空间关系。近期的方法试图依托文本描述的语义来解决这些局限，即利用大语言模型（Large Language Models, LLMs）将描述拆解为若干问答子集。但这些方法主要在表层运作，未能融入更深层的词汇理解，同时还会引入LLM生成的错误假设。针对这些问题，我们提出了“带矛盾与蕴涵的字幕扩展”（Caption Expansion with Contradictions and Entailments, CECE），这是一种利用自然语言推理（Natural Language Inference, NLI）从给定前提生成蕴涵句和矛盾句的系统性方法。CECE能生成词汇层面多样的句子，同时保留其核心语义。通过大量实验，我们表明CECE提升了可解释性，减少了对有偏或表层特征的过度依赖。通过将CECE与原始前提进行平衡，我们无需额外微调就实现了对先前方法的显著提升，在衡量图文对齐与人类判断一致性的基准上取得了当前最优结果，在Winoground的组得分上比此前最优工作（经针对性数据微调）提升+19.2%，在EqBen的组得分上提升+12.9%。
英文摘要：
Compositional reasoning in Vision-Language Models (VLMs) remains challenging as these models often struggle to relate objects, attributes, and spatial relationships. Recent methods aim to address these limitations by relying on the semantics of the textual description, using Large Language Models (LLMs) to break them down into subsets of questions and answers. However, these methods primarily operate on the surface level, failing to incorporate deeper lexical understanding while introducing incorrect assumptions generated by the LLM. In response to these issues, we present Caption Expansion with Contradictions and Entailments (CECE), a principled approach that leverages Natural Language Inference (NLI) to generate entailments and contradictions from a given premise. CECE produces lexically diverse sentences while maintaining their core meaning. Through extensive experiments, we show that CECE enhances interpretability and reduces overreliance on biased or superficial features. By balancing CECE along the original premise, we achieve significant improvements over previous methods without requiring additional fine-tuning, producing state-of-the-art results on benchmarks that score agreement with human judgments for image-text alignment, and achieving an increase in performance on Winoground of +19.2% (group score) and +12.9% on EqBen (group score) over the best prior work (finetuned with targeted data).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
