SpatialMosaic：一个多视角VLM数据集用于部分可见性
SpatialMosaic: A Multiview VLM Dataset for Partial Visibility
AI总结：
本文提出SpatialMosaic数据集，通过多视角图像生成2M问答对，引入SpatialMosaic-Bench基准测试，结合3D重建模型的混合框架提升空间推理能力。
摘要：
多模态大语言模型（MLLMs）的快速发展解锁了增强3D场景理解和空间推理的潜力。最近的研究探索了从多视角图像直接学习空间推理，使MLLMs无需显式3D重建即可理解3D场景。然而，现实环境中常见的挑战，如部分可见性、遮挡和低重叠条件，仍需进一步研究。为解决这些限制，我们提出一个可扩展的多视角数据生成和标注流程，构建出现实的空间推理问答，从而产生SpatialMosaic，一个包含2M问答对的全面指令微调数据集。我们进一步引入SpatialMosaic-Bench，一个用于评估多视角空间推理的挑战性基准，包含1M问答对，涵盖6个任务。我们的数据集涵盖室内外场景，能够全面评估多样化的现实场景。此外，我们引入了一个新的多视角基线，即SpatialMosaicVLM，一种混合框架，将3D重建模型作为几何编码器整合到VLMs中，以实现稳健的空间推理。大量实验表明，我们提出的数据集有效提升了在挑战性多视角条件下的空间推理能力，验证了我们数据生成流程在构建现实且具有挑战性的问答中的有效性。代码和数据集将很快发布。
英文摘要：
Recent progress in Multimodal Large Language Models (MLLMs) has enabled 3D scene understanding and spatial reasoning directly from multi-view images, without requiring explicit 3D reconstructions. Nevertheless, key challenges that frequently arise in real-world environments, such as partial visibility, occlusion, and low-overlap conditions that require reasoning from fragmented visual cues, remain under-explored. To address these limitations, we propose a scalable multi-view data generation and annotation pipeline that constructs realistic spatial reasoning QAs, resulting in SpatialMosaic, a comprehensive instruction-tuning dataset with 2M QA pairs. We further introduce SpatialMosaic-Bench, a challenging benchmark for evaluating multi-view spatial reasoning under complex and diverse scenarios, consisting of 1M QA pairs across 11 tasks with both multiple-choice and numerical-answer formats. Our dataset spans both indoor and outdoor scenes, enabling comprehensive evaluation across diverse real-world scenarios. In addition, we provide a practical baseline for multi-view settings by integrating geometry encoders into VLMs for improved cross-view consistency and spatial grounding. Extensive experiments demonstrate that our dataset effectively enhances spatial reasoning under challenging multi-view conditions, validating the effectiveness of our data generation pipeline in constructing realistic and challenging QAs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
