SIRI-Bench: 通过复杂推理任务挑战VLMs的空间智能
SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Tasks
AI总结：
SIRI-Bench通过空间接地推理任务评估VLMs的空间结构智能，包含9000个视频问题答案三元组，实验表明先进VLMs在结构空间推理上面临挑战。
摘要：
大型语言模型（LLMs）在复杂推理任务上的强化学习推动了快速发展。相比之下，空间智能对视觉语言模型（VLMs）在现实交互中至关重要，但其复杂空间推理的系统研究仍不足。为此，我们引入SIRI-Bench基准，通过空间接地推理任务评估VLMs的结构空间智能。SIRI-Bench包含9000个视频问题答案三元组，每个问题嵌入在现实3D场景中。该基准精心设计，解决每个问题需要空间理解和结构推理。为促进大规模数据合成，我们开发了自动场景创建引擎，利用协作LLM代理将抽象数学问题转化为忠实的3D场景。实验结果表明，最先进的VLMs在SIRI-Bench上表现显著不足，凸显了结构空间推理的挑战。我们希望本研究能引起研究人员对空间接地推理的关注，并推动VLMs在视觉问题解决中的发展。
英文摘要：
Large Language Models (LLMs) have undergone rapid progress, largely attributed to reinforcement learning on complex reasoning tasks. In contrast, while spatial intelligence is fundamental for Vision-Language Models (VLMs) in real-world interaction, the systematic study of their complex spatial reasoning remains underexplored. To bridge this gap, we introduce SIRI-Bench, a benchmark designed to evaluate VLMs' structural spatial intelligence through spatial-grounded reasoning tasks. SIRI-Bench comprises 9,000 video-question-answer triplets, where each problem is embedded in a realistic 3D scene. The benchmark is carefully designed so that solving each problem requires both spatial comprehension and structural reasoning. To facilitate large-scale data synthesis, we develop an Automatic Scene Creation Engine that employs collaborative LLM agents to translate abstract mathematical problems into faithful 3D scenes. Experimental results reveal that state-of-the-art VLMs struggle significantly on SIRI-Bench, underscoring the challenge of structural spatial reasoning. We hope that our study will bring researchers' attention to spatially grounded reasoning and advance VLMs in visual problem-solving.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
