视觉问答中视觉语言模型的敏感性分析
Analyzing the Sensitivity of Vision Language Models in Visual Question Answering
AI总结：
本研究以格莱斯对话合作原则为视角，通过在VQA v2.0数据集问题中添加修饰语，测试GPT-4o等3个先进VLM对准则违反的敏感度，发现修饰语会导致模型性能持续下降，为理解VLM局限性提供了新方向。
摘要：
我们可以将视觉问答视为人类与AI系统之间的（多模态）对话。本研究以格莱斯（Grice）提出的对话合作原则为视角，探究视觉语言模型（Vision Language Models, VLMs）的敏感性。具体而言，即便格莱斯对话准则被违反，人类通常仍能理解对话内容，只是需要付出更多认知努力。本研究考察VLMs是否能以与人类相似的方式应对格莱斯准则的违反情况。具体来说，我们在人工设计的问题中添加修饰语，分析VLMs对这些修饰语的响应。我们选用3个当前最先进的VLM——GPT-4o、Claude-3.5-Sonnet和Gemini-1.5-Flash，基于VQA v2.0数据集的问题开展实验。初步结果显示，随着修饰语的加入，VLMs的性能持续下降，这表明本研究方法是理解VLMs局限性的一个有前景的方向。
英文摘要：
We can think of Visual Question Answering as a (multimodal) conversation between a human and an AI system. Here, we explore the sensitivity of Vision Language Models (VLMs) through the lens of cooperative principles of conversation proposed by Grice. Specifically, even when Grice's maxims of conversation are flouted, humans typically do not have much difficulty in understanding the conversation even though it requires more cognitive effort. Here, we study if VLMs are capable of handling violations to Grice's maxims in a manner that is similar to humans. Specifically, we add modifiers to human-crafted questions and analyze the response of VLMs to these modifiers. We use three state-of-the-art VLMs in our study, namely, GPT-4o, Claude-3.5-Sonnet and Gemini-1.5-Flash on questions from the VQA v2.0 dataset. Our initial results seem to indicate that the performance of VLMs consistently diminish with the addition of modifiers which indicates our approach as a promising direction to understand the limitations of VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
