arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.05382cs.CVcs.LG
收藏

VRIQ:评估和分析视觉推理IQ的VLMs基准测试

VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models

University of Southern California, Los Angeles, USA(美国南加州大学)

Tina Khezresmaeilzadeh, Jike Zhong, Konstantinos Psounis

AI总结:

VRIQ基准测试评估了VLMs的视觉推理能力,发现其在抽象推理任务中表现薄弱,主要源于感知限制,提出细粒度诊断方法以改进多模态系统中的视觉推理。

摘要:

最近在视觉语言模型(VLMs)上的进展引发了关于它们是否能够可靠地进行非语言推理的问题。为此,我们介绍了VRIQ(视觉推理IQ),一个全新的基准测试,旨在评估和分析VLMs的视觉推理能力。我们对两种任务集进行了模型评估:抽象谜题式和自然图像推理任务。我们发现,在抽象谜题中,性能接近随机,平均准确率约为28%,而在自然任务中,结果更好但仍然薄弱,准确率为45%。我们还发现,工具增强的推理仅显示出适度的改进。为了揭示这种弱点的来源,我们引入了诊断探针,针对感知和推理进行分析。我们的分析表明,大约56%的失败源于感知本身,43%源于感知和推理,只有1%源于推理本身。这促使我们设计细粒度的诊断探针问题,针对特定的感知类别(例如,形状、计数、位置、3D/深度),揭示某些类别导致更多的失败。我们的基准测试和分析表明,当前的VLMs,即使有视觉推理工具,仍然不可靠的抽象推理者,主要由于感知限制,并为多模态系统中的视觉推理改进提供了原则性的基础。

英文摘要:

Vision-language model (VLM) errors on visual reasoning may come from missing visual structure, from mistakes after that structure is extracted, or from both. We introduce VRIQ, a benchmark of 1,391 human-verified IQ-style visual reasoning puzzles spanning five psychometric categories, each instantiated in category-matched abstract and natural-image domains. To study perception and reasoning more directly, we also introduce DiagVRIQ, a 209-item diagnostic subset paired with human-verified, rule-neutral textual descriptions. Across 21 VLMs, the 17 non-frontier models reach a 26.8% unweighted category mean on abstract puzzles, while three 2026 frontier models improve substantially but unevenly, with unweighted category means of 41.3--70.9% on abstract puzzles and at most 78.2% on natural-image puzzles; all remain below human performance. On DiagVRIQ, even when their descriptions are judged correct, the strongest models still answer approximately 9--12% of those items incorrectly. Two interventions, ground-truth description augmentation and tool-augmented inference, improve accuracy, with the largest tool gains on spatially demanding categories, but neither eliminates errors. VRIQ and DiagVRIQ provide a framework for studying visual reasoning through complementary end-to-end, description-based, and intervention-based analyses rather than treating it as a single monolithic capability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑