arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.04778cs.CV
收藏

MM-R$^3$:关于视觉-语言模型(VLMs)的(不)一致性

MM-R$^3$: On (In-)Consistency of Vision-Language Models (VLMs)

University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(Vector人工智能研究所)

Shih-Han Chou, Shivam Chandhok, James J. Little, Leonid Sigal

AI总结:

本文提出MM-R3基准,分析视觉语言模型在一致性与准确率上的表现,并提出适配器模块缓解不一致性,在BLIP-2和LLaVa 1.5M上实现一致性显著提升。

摘要:

随着LLMs及其变体的出现,涌现了大量研究分析此类模型在一系列任务上的性能。尽管大多数研究侧重于通过任务准确率(如视觉问答、定位)来评估最先进的视觉语言模型的能力,但我们的工作探索了相关但互补的方面——一致性,即VLM对语义相似的查询产生语义相似或相同响应的能力。我们指出,一致性是VLM鲁棒性和可信度的基本前提(必要但非充分条件)。基于这一视角,我们提出了MM-R3基准,使我们能够在问题重述、图像重风格化和上下文推理这三个任务上分析SoTA VLM在一致性和准确率方面的性能。我们的分析揭示,一致性并不总是与准确率对齐,表明准确率较高的模型不一定更一致,反之亦然。此外,我们提出了一种简单而有效的缓解策略,形式为一个训练用于最小化提示间不一致性的适配器模块。通过该策略,我们在BLIP-2和LLaVa 1.5M等广泛使用的VLM上,平均实现了一致性5.7%和12.5%的绝对提升。

英文摘要:

With the advent of LLMs and variants, a flurry of research has emerged, analyzing the performance of such models across an array of tasks. While most studies focus on evaluating the capabilities of state-of-the-art (SoTA) Vision Language Models (VLMs) through task accuracy (e.g., visual question answering, grounding), our work explores the related but complementary aspect of consistency - the ability of a VLM to produce semantically similar or identical responses to semantically similar queries. We note that consistency is a fundamental prerequisite (necessary but not sufficient condition) for robustness and trust in VLMs. Armed with this perspective, we propose the MM-R3 benchmark, which allows us to analyze performance, in terms of consistency and accuracy, of SoTA VLMs on three tasks: Question Rephrasing, Image Restyling, and Context Reasoning. Our analysis reveals that consistency does not always align with accuracy, indicating that models with higher accuracy are not necessarily more consistent, and vice versa. Furthermore, we propose a simple yet effective mitigation strategy in the form of an adapter module trained to minimize inconsistency across prompts. With our proposed strategy, we are able to achieve absolute improvements of 5.7% and 12.5%, on average on widely used VLMs such as BLIP-2 and LLaVa 1.5M in terms of consistency over their existing counterparts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑