VISLA Benchmark：评估嵌入对语义和词汇变化的敏感性
VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations
AI总结：
本文提出VISLA基准，通过三向语义（不）等价任务评估34个VLMs和20个ULMs对语义和词汇变化的敏感性，发现VLMs文本编码器比单模态模型更敏感，揭示了模型在区分词汇与语义变化上的困难。
摘要：
尽管取得了显著成功，最先进的语言模型在掌握某些重要语义细节方面仍面临挑战。本文介绍了VISLA（Variance and Invariance to Semantic and Lexical Alterations）基准，旨在评估语言模型的语义和词汇理解能力。VISLA提出了一个三向语义（不）等价任务，包含与图像关联的三元组句子，以评估视觉语言模型（VLMs）和单模态语言模型（ULMs）。对34个VLMs和20个ULMs的评估揭示了在区分词汇和语义变化方面令人惊讶的困难。语言模型编码的空间语义似乎对词汇信息也高度敏感。值得注意的是，VLMs的文本编码器比单模态文本编码器对语义和词汇变化表现出更大的敏感性。我们的贡献包括统一图像到文本和文本到文本检索任务、无需微调的即用型评估，以及在词汇变化存在时评估LMs的语义（不）变性。结果突出了各种视觉和单模态语言模型的优缺点，有助于更深入理解它们的能力。数据和代码将发布于https://github.com/Sri-Harsha/visla_benchmark。
英文摘要：
Despite their remarkable successes, state-of-the-art language models face challenges in grasping certain important semantic details. This paper introduces the VISLA (Variance and Invariance to Semantic and Lexical Alterations) benchmark, designed to evaluate the semantic and lexical understanding of language models. VISLA presents a 3-way semantic (in)equivalence task with a triplet of sentences associated with an image, to evaluate both vision-language models (VLMs) and unimodal language models (ULMs). An evaluation involving 34 VLMs and 20 ULMs reveals surprising difficulties in distinguishing between lexical and semantic variations. Spatial semantics encoded by language models also appear to be highly sensitive to lexical information. Notably, text encoders of VLMs demonstrate greater sensitivity to semantic and lexical variations than unimodal text encoders. Our contributions include the unification of image-to-text and text-to-text retrieval tasks, an off-the-shelf evaluation without fine-tuning, and assessing LMs' semantic (in)variance in the presence of lexical alterations. The results highlight strengths and weaknesses across diverse vision and unimodal language models, contributing to a deeper understanding of their capabilities. % VISLA enables a rigorous evaluation, shedding light on language models' capabilities in handling semantic and lexical nuances. Data and code will be made available at https://github.com/Sri-Harsha/visla_benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
