arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.21902cs.CV
收藏

眼见为实:评估与缓解多模态大语言模型(MLLMs)中语言先验对视觉感知的干扰

Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs

School of Computer Science and Engineering(计算机科学与工程学院);University of Electronic Science and Technology of China(电子科技大学)

Li Ma, Yan Zhuang, Jiawen Deng, Fuji Ren

AI总结:

该研究针对MLLMs中语言先验干扰视觉感知的问题,构建ViSiP-Bench基准并提出MAPS方法,评估发现先验依附普遍且不随规模提升改善,MAPS可弱化过度先验依赖且不影响通用能力。

摘要:

多模态大语言模型(MLLMs)中的语言先验会覆盖视觉证据,损害视觉感知能力。现有基准假设不同模型间存在共享先验,且仅依赖准确率指标,但跨模型先验一致性低于80%,超过15%的测试项无需图像即可答对,导致准确率高估了模型对先验的抵抗能力。为解决这些局限,我们推出ViSiP-Bench,该基准基于六大先验类别下的罕见但真实场景,配备模型感知的动态先验诱导协议,以及三项比准确率更精准、更连续地测量先验抵抗能力的指标。对24个模型的评估显示,先验依附现象普遍存在,且不随模型规模提升而改善;较大模型往往要么强烈覆盖自身先验,要么坚定依附于先验。因此,我们提出MAPS方法,该方法识别每个模型的最强先验,并基于事实依据的知识进行微调,以在不破坏模型通用能力的前提下,弱化过度的先验依赖。

英文摘要:

Language priors in MLLMs can override visual evidence, impairing visual perception. Existing benchmarks assume shared priors across models and rely solely on accuracy. However, cross-model prior agreement falls below 80%, and over 15% of items can be answered correctly without any image, causing accuracy to overestimate prior resistance. To address these limitations, we introduce ViSiP-Bench, a benchmark grounded in rare-but-real scenarios across six prior categories, with a model-aware dynamic prior induction protocol and three metrics that more precisely and continuously measure prior resistance than accuracy alone. Evaluating 24 models reveals that prior adherence is pervasive, does not improve with scale, and larger models tend to either strongly override their priors or firmly adhere to them. We therefore propose MAPS, which identifies each model's strongest priors and fine-tunes on factually grounded knowledge to soften excessive prior reliance without disrupting general capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑