生成式视觉语言模型对语义和词汇改动提示的敏感性
Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts
AI总结：
本文利用 SugarCrepe++ 数据集研究生成式视觉语言模型对提示中词汇和语义改动的敏感性，发现模型对无语义变化的词汇改动高度敏感，且该脆弱性会损害输出一致性技术的性能。
摘要：
尽管面向生成式视觉语言模型（VLMs）的提示微调技术大量涌现，这些模型对提示中词汇和语义改动的敏感程度仍不清楚。本文使用 SugarCrepe++ 数据集评估生成式 VLMs 理解文本中词汇和语义变化的能力。我们分析了 VLMs 对未伴随语义变化的提示词汇改动的敏感性。研究结果表明，生成式 VLMs 对此类改动高度敏感。此外，我们表明这一脆弱性会影响旨在实现其输出一致性的技术的性能。
英文摘要：
Despite the significant influx of prompt-tuning techniques for generative vision-language models (VLMs), it remains unclear how sensitive these models are to lexical and semantic alterations in prompts. In this paper, we evaluate the ability of generative VLMs to understand lexical and semantic changes in text using the SugarCrepe++ dataset. We analyze the sensitivity of VLMs to lexical alterations in prompts without corresponding semantic changes. Our findings demonstrate that generative VLMs are highly sensitive to such alterations. Additionally, we show that this vulnerability affects the performance of techniques aimed at achieving consistency in their outputs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
