GamiBench: 通过折纸任务评估多模态大语言模型的空间推理和2D到3D规划能力
GamiBench: Evaluating Spatial Reasoning and 2D-to-3D Planning Capabilities of MLLMs with Origami Folding Tasks
AI总结:
GamiBench通过折纸任务评估多模态大语言模型的空间推理和2D到3D规划能力,引入新指标并揭示模型在复杂折叠任务中的不足。
摘要:
多模态大语言模型(MLLMs)在感知和遵循指令方面表现出色,但在空间推理方面仍存在困难:即能够精神上跟踪和在多个视角和时间上操纵物体的能力。空间推理是人类智能的关键组成部分,但大多数现有基准测试集中在静态图像或最终输出上,未能考虑这种技能的序列性和视角依赖性。为弥补这一差距,我们介绍了GamiBench,这是一个旨在通过折纸启发的折叠任务评估MLLMs的空间推理和2D到3D规划能力的基准测试。GamiBench包括186个常规和186个不可能的2D折痕模式,配以对应的3D折叠形状,这些由三个视觉问答(VQA)任务中的六个不同视角产生:预测3D折叠配置、区分有效视角和检测不可能的模式。与以往只评估最终预测的基准测试不同,GamiBench全面评估整个推理过程——测量跨视角一致性、通过不可能折叠检测测量物理可行性以及解释中间折叠步骤的解读。它进一步引入了新的诊断指标——视角一致性(VC)和不可能折叠选择率(IFSR)——以衡量模型如何处理不同复杂度的折叠。我们的实验表明,即使领先模型如GPT-5和Gemini-2.5-Pro在单步空间理解上也表现不佳。这些贡献建立了一个标准化框架,用于评估MLLMs的几何理解和空间推理能力。数据集和代码:https://github.com/stvngo/GamiBench。
英文摘要:
Multimodal large language models (MLLMs) are proficient in perception and instruction-following, but they still struggle with spatial reasoning: the ability to mentally track and manipulate objects across multiple views and over time. Spatial reasoning is a key component of human intelligence, but most existing benchmarks focus on static images or final outputs, failing to account for the sequential and viewpoint-dependent nature of this skill. To close this gap, we introduce GamiBench, a benchmark designed to evaluate spatial reasoning and 2D-to-3D planning in MLLMs through origami-inspired folding tasks. GamiBench includes 186 regular and 186 impossible 2D crease patterns paired with their corresponding 3D folded shapes, produced from six distinct viewpoints across three visual question-answering (VQA) tasks: predicting 3D fold configurations, distinguishing valid viewpoints, and detecting impossible patterns. Unlike previous benchmarks that assess only final predictions, GamiBench holistically evaluates the entire reasoning process--measuring cross-view consistency, physical feasibility through impossible-fold detection, and interpretation of intermediate folding steps. It further introduces new diagnostic metrics--viewpoint consistency (VC) and impossible fold selection rate (IFSR)--to measure how well models handle folds of varying complexity. Our experiments show that even leading models such as GPT-5 and Gemini-2.5-Pro struggle on single-step spatial understanding. These contributions establish a standardized framework for evaluating geometric understanding and spatial reasoning in MLLMs. Dataset and code: https://github.com/stvngo/GamiBench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。