MindEdit-Bench：基于野外照片的VLM中对象级反事实空间推理基准
MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos
AI总结：
提出MindEdit-Bench基准，包含六项空间推理任务，其中两项新任务（L4和L5）测试对象级反事实推理，VLM平均准确率仅8%-31%，远低于人类的81%-97%。
摘要：
视觉语言模型（VLM）的基准测试主要测试观察性空间推理：模型描述输入中已可见的关系。现有的假设性任务通常保持场景固定而改变观察者。VLM能否预测假设移动或旋转物体的后果？我们引入了MindEdit-Bench，这是一个包含六项空间推理任务的基准，通过自动野外3D场景图提取流水线，从新拍摄的室内场景的三张照片智能手机三连拍中构建。四项任务探测对观察结构的感知和视角变换；两项新任务L4（空间编辑）和L5（跨视图可见性编辑）测试对象级反事实推理，其中正确答案在所有输入图像中均不存在。每个问题提供8-24个结构化答案选项，实现答案字母级别的空间和回退错误诊断。该基准涵盖120个私有室内场景，不来自公共数据集，减少了公共数据预训练重叠风险。在15个VLM上对1003个人工验证问题的测试中，任务平均VLM准确率仅为8%-31%，而人类多数投票准确率为81%-97%。人类与最佳VLM之间的汇总差距为53个百分点，每项任务至少39个百分点。结构化答案空间进一步揭示了非均匀失败，包括较弱的相机深度轴推理和在困难可见性编辑案例上的回退行为。
英文摘要：
Benchmarks for vision-language models (VLMs) mostly test observational spatial reasoning: models describe relations already visible in the input. Existing what-if tasks typically vary the observer while keeping the scene fixed. Can VLMs instead predict the consequences of hypothetically moving or rotating an object? We introduce MindEdit-Bench, a benchmark of six spatial reasoning tasks built from three-photo smartphone triplets of newly captured indoor scenes via an automatic in-the-wild 3D scene-graph extraction pipeline. Four tasks probe perception and perspective transformation over observed structure; two new tasks, L4 (spatial editing) and L5 (cross-view visibility editing), probe object-level counterfactual reasoning, where correct answers are absent from all input images. Each question provides 8-24 structured answer choices, enabling answer-letter-level diagnosis of spatial and fallback errors. The benchmark covers 120 private indoor scenes not drawn from public datasets, reducing public-data pretraining-overlap risk. Across 15 VLMs on 1,003 human-verified questions, task-wise mean VLM accuracy is only 8%-31%, versus 81%-97% human majority-vote accuracy. The pooled human--best-VLM gap is 53 pp, with at least 39 pp on every task. The structured answer space further reveals non-uniform failures, including weaker camera-depth-axis inference and fallback behavior on difficult visibility-editing cases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
