arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.09430cs.CV
收藏

Video-MSR: 多跳空间推理能力的MLLMs基准测试

Video-MSR: Benchmarking Multi-hop Spatial Reasoning Capabilities of MLLMs

Baidu Inc.(百度公司);Nanjing University(南京大学);The University of Queensland(昆士兰大学);Peking University(北京大学)

Rui Zhu, Xin Shen, Shuchen Wu, Chenxi Miao, Xin Yu, Yang Li, Weikang Li, Deguo Xia, Jizhou Huang

AI总结:

Video-MSR通过四个任务评估MLLMs的多跳空间推理能力,发现模型在复杂推理中存在显著不足,并通过微调提升性能。

摘要:

空间推理已成为多模态大语言模型(MLLMs)的关键能力,受到越来越多的关注和快速的发展。然而,现有基准测试主要集中在单步感知到判断的任务上,留下需要复杂视觉空间逻辑链的场景显著未被探索。为填补这一空白,我们引入Video-MSR,第一个专门设计用于评估动态视频场景中多跳空间推理(MSR)的基准测试。Video-MSR通过四个不同的任务系统地探测MSR能力:受限定位、基于链式参考检索、路线规划和反事实物理推断。我们的基准测试包含3,052个高质量视频实例,4,993个问题-答案对,通过结合先进模型生成与严格的人类验证的可扩展、视觉基础的管道构建。通过全面评估20种最先进的MLLMs,我们发现显著的局限性,揭示尽管模型在表层感知方面表现出色,但在MSR任务中表现出明显的性能下降,经常在多步推断中出现空间迷失和幻觉。为缓解这些不足并增强模型的MSR能力,我们进一步整理了MSR-9K,一个专门的指令微调数据集,并对Qwen-VL进行微调,实现了在Video-MSR上的+7.82%的绝对提升。我们的结果强调了多跳空间指令数据的有效性,并确立了Video-MSR作为未来研究的基础。代码和数据将在https://github.com/ruiz-nju/Video-MSR上提供。

英文摘要:

Spatial reasoning has emerged as a critical capability for Multimodal Large Language Models (MLLMs), drawing increasing attention and rapid advancement. However, existing benchmarks primarily focus on single-step perception-to-judgment tasks, leaving scenarios requiring complex visual-spatial logical chains significantly underexplored. To bridge this gap, we introduce Video-MSR, the first benchmark specifically designed to evaluate Multi-hop Spatial Reasoning (MSR) in dynamic video scenarios. Video-MSR systematically probes MSR capabilities through four distinct tasks: Constrained Localization, Chain-based Reference Retrieval, Route Planning, and Counterfactual Physical Deduction. Our benchmark comprises 3,052 high-quality video instances with 4,993 question-answer pairs, constructed via a scalable, visually-grounded pipeline combining advanced model generation with rigorous human verification. Through a comprehensive evaluation of 20 state-of-the-art MLLMs, we uncover significant limitations, revealing that while models demonstrate proficiency in surface-level perception, they exhibit distinct performance drops in MSR tasks, frequently suffering from spatial disorientation and hallucination during multi-step deductions. To mitigate these shortcomings and empower models with stronger MSR capabilities, we further curate MSR-9K, a specialized instruction-tuning dataset, and fine-tune Qwen-VL, achieving a +7.82% absolute improvement on Video-MSR. Our results underscore the efficacy of multi-hop spatial instruction data and establish Video-MSR as a vital foundation for future research. The code and data will be available at https://github.com/ruiz-nju/Video-MSR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑