arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2601.04897cs.CLcs.CVcs.LGcs.MM
收藏

V-FAT:基于文本偏见的视觉真实性基准测试

V-FAT: Benchmarking Visual Fidelity Against Text-bias

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Meituan(美团);University of Oxford(牛津大学)

Ziteng Wang, Yujie He, Guanliang Li, Siqi Yang, Jiaqi Xiong, Songxiang Liu

AI总结:

V-FAT通过三级评估框架量化文本偏见对视觉真实性的干扰,揭示多模态大语言模型在高语言主导性下的视觉崩溃问题。

摘要:

多模态大语言模型(MLLMs)近期的进展在标准视觉推理基准测试中展示了令人印象深刻的性能。然而,人们越来越担心这些模型过度依赖语言捷径而非真正的视觉基础,这种现象我们称之为文本偏见。在本文中,我们探讨了视觉感知与语言先验之间的根本张力。我们将这种偏见的来源分为两个维度:内部语料偏见,源于预训练中的统计相关性,以及外部指令偏见,源于对齐引起的顺从倾向。为了量化这种影响,我们引入了V-FAT(视觉真实性对抗文本偏见)基准测试,包含来自六个语义领域的4026个VQA实例。V-FAT采用三级评估框架,系统地增加视觉证据与文本信息之间的冲突:(L1)来自非常规图像的内部偏见,(L2)来自误导性指令的外部偏见,以及(L3)两者同时存在的协同偏见。我们引入了视觉鲁棒性分数(VRS),一种旨在惩罚“幸运”的语言猜测并奖励真正的视觉真实性的指标。对12个前沿MLLMs的评估显示,尽管模型在现有基准测试中表现出色,但在高语言主导性下却经历了显著的视觉崩溃。

英文摘要:

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive performance on standard visual reasoning benchmarks. However, there is growing concern that these models rely excessively on linguistic shortcuts rather than genuine visual grounding, a phenomenon we term Text Bias. In this paper, we investigate the fundamental tension between visual perception and linguistic priors. We decouple the sources of this bias into two dimensions: Internal Corpus Bias, stemming from statistical correlations in pretraining, and External Instruction Bias, arising from the alignment-induced tendency toward sycophancy. To quantify this effect, we introduce V-FAT (Visual Fidelity Against Text-bias), a diagnostic benchmark comprising 4,026 VQA instances across six semantic domains. V-FAT employs a Three-Level Evaluation Framework that systematically increases the conflict between visual evidence and textual information: (L1) internal bias from atypical images, (L2) external bias from misleading instructions, and (L3) synergistic bias where both coincide. We introduce the Visual Robustness Score (VRS), a metric designed to penalize "lucky" linguistic guesses and reward true visual fidelity. Our evaluation of 12 frontier MLLMs reveals that while models excel in existing benchmarks, they experience significant visual collapse under high linguistic dominance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑