SUGARCREPE++ Dataset: 视觉-语言模型对语义与词汇变更的敏感性
SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations
AI总结：
该研究提出SUGARCREPE++数据集，用于分析视觉与单模态语言模型对词汇和语义变更的敏感性，实验发现现有模型在区分相关变化上存在困难，为视觉-语言领域带来新挑战。
摘要：
尽管包括视觉-语言模型（VLM）和单模态语言模型（ULM）在内的最先进大语言模型（LLM）取得了显著成功，但它们仍无法理解精确语义。例如，用不同词汇构成表达的语义等价句子会引发差异极大的表征，而这种差异的程度及其对编码语义的影响目前尚不十分明确。本文提出SUGARCREPE++数据集，用于分析VLM和ULM对词汇与语义变更的敏感性。SUGARCREPE++数据集中的每个样本包含一张图像和对应的三元字幕组：一对语义等价但词汇不同的正例字幕，以及一个难负例字幕，这给语言模型提出了一个三分类语义（不）等价问题。我们全面评估了架构、预训练目标和数据集各不相同的VLM和ULM，以此在SUGARCREPE++数据集上开展性能基准测试。实验结果凸显了VLM在区分词汇与语义变化方面存在困难，尤其是在物体属性和空间关系方面。尽管使用更大预训练数据集、模型规模更大且具备多种预训练目标的VLM在SUGARCREPE++上表现更好，但仍有很大的提升空间。我们发现，所有在组合性数据集上表现更好的模型，在SUGARCREPE++上的表现未必同样出色，这表明仅靠组合性可能不足以理解语义和词汇变更。鉴于SUGARCREPE++数据集所针对的特性的重要性，它为视觉-语言领域提出了一项新挑战。
英文摘要：
Despite their remarkable successes, state-of-the-art large language models (LLMs), including vision-and-language models (VLMs) and unimodal language models (ULMs), fail to understand precise semantics. For example, semantically equivalent sentences expressed using different lexical compositions elicit diverging representations. The degree of this divergence and its impact on encoded semantics is not very well understood. In this paper, we introduce the SUGARCREPE++ dataset to analyze the sensitivity of VLMs and ULMs to lexical and semantic alterations. Each sample in SUGARCREPE++ dataset consists of an image and a corresponding triplet of captions: a pair of semantically equivalent but lexically different positive captions and one hard negative caption. This poses a 3-way semantic (in)equivalence problem to the language models. We comprehensively evaluate VLMs and ULMs that differ in architecture, pre-training objectives and datasets to benchmark the performance of SUGARCREPE++ dataset. Experimental results highlight the difficulties of VLMs in distinguishing between lexical and semantic variations, particularly in object attributes and spatial relations. Although VLMs with larger pre-training datasets, model sizes, and multiple pre-training objectives achieve better performance on SUGARCREPE++, there is a significant opportunity for improvement. We show that all the models which achieve better performance on compositionality datasets need not perform equally well on SUGARCREPE++, signifying that compositionality alone may not be sufficient for understanding semantic and lexical alterations. Given the importance of the property that the SUGARCREPE++ dataset targets, it serves as a new challenge to the vision-and-language community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
