IDEAL-Bench：用于分析3D布局推理的室内数据集和评估套件
IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning
AI总结：
介绍IDEAL-Bench评估套件，让视觉语言模型预测室内场景结构化3D布局，基于IDEAL-Scenes数据集，评估15个模型发现任务待解、模型存在不对称性及排名差异，为下一代模型空间智能评估铺路。
摘要：
空间问答是评估视觉语言模型空间智能的主要范式，但整体3D布局推理这一空间能力轴评估不足。为此引入IDEAL-Bench评估套件，在10种室内场景要求模型预测结构化3D布局并打分。基于IDEAL-Scenes数据集评估15个模型有三项发现，确立其为诊断套件，为下一代模型空间智能评估铺路。
英文摘要：
Spatial question answering is the dominant paradigm for evaluating spatial intelligence in Vision-Language Models (VLMs), but it leaves a complementary axis of spatial competence under-evaluated: holistic 3D layout inference, which predicts every visible object's pose and extent from a single image in a structured form. To this end, we introduce IDEAL-Bench, an evaluation suite that requires VLMs to predict structured 3D layouts on photorealistic indoor scenes across 10 room types, scored along five numerical dimensions and a perceptual render-and-compare protocol. By operating on semantically realistic scenes with full asset substitution under controlled lighting and viewpoint, IDEAL-Bench moves beyond CLEVR-style simple geometric primitives so that any image-space discrepancy reflects spatial reasoning alone. The benchmark is built on IDEAL-Scenes, a procedurally generated dataset of 1,000 re-renderable Blender environments with ground-truth layouts. Evaluating 15 prominent VLMs reveals three findings: the task remains substantially unsolved, with the strongest model reaching only 62.1/100 overall; all models exhibit a sharp asymmetry between object recognition and geometric regression, indicating that current VLMs are trained to describe scenes rather than to measure them; model rankings partially diverge from those on QA-based and primitive-reconstruction benchmarks: top-tier consensus holds, but mid-tier rankings shift substantially. Collectively, these findings establish IDEAL-Bench as a diagnostic suite, targeting the geometric and structural competencies that QA-based evaluation cannot surface, and paving the way towards more rigorous evaluation of spatial intelligence in next-generation VLMs. Together, these findings position IDEAL-Bench as a principled diagnostic for whether future VLMs achieve genuine spatial understanding rather than linguistic approximations of it.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
