arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.26659cs.CVcs.CL
收藏

SpatialScout:VLM智能体能否主动探索并绘制3D场景地图?

SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?

Peking University(北京大学);Tianjin University(天津大学)

Zhaoxi Zhang, Hao Liang, Linxuan Wu, Yiming Xu, Yifu Yuan, Kun Liang, Chenming Tang, Meiyi Qiang, Wentao Zhang, Weikang Li

AI总结:

提出SpatialScout基准评估VLM智能体的主动对象级3D地图绘制能力,实验显示最强模型Claude Opus 5.5仅达41.71%平均重叠分,暴露了构建精确3D地图的显著差距。

摘要:

空间智能要求智能体主动获取视觉证据,并将观测结果整合为三维世界的连贯表示。然而空间问答仅能提供有限证据,无法证明模型是否能通过自主探索构建精确的度量地图。我们提出SpatialScout,这是一个用于评估视觉语言模型(VLM)智能体在主动对象级3D地图绘制任务中的基准。SpatialScout包含80个多样化的室内场景,共含3140个目标对象。给定对象清单、RGB观测值、相机标定参数和相机位姿,智能体控制相机探索每个场景,并在共享坐标系中估计对象中心。该任务要求在有限的交互预算下协调视点选择、跨视图关联、几何估计和地图修正。评估通过对象级重叠分数联合衡量定位精度和目标覆盖率,该分数会惩罚缺失预测。对前沿VLM的实验显示出显著局限性:评估中最强的模型Claude Opus 5.5的平均重叠分数为41.71%,其中47.23%的目标超过交并比(IoU)阈值0.5,仅3.25%超过0.9。这些结果表明,恢复粗略对象布局与构建完整精确的度量地图之间存在显著差距。SpatialScout为开发通过主动感知构建可靠3D空间表示的智能体提供了定量测试平台。代码可在此https URL获取。

英文摘要:

Spatial intelligence requires agents to actively acquire visual evidence and integrate observations into a coherent representation of the three-dimensional world. Yet spatial question answering offers limited evidence of whether models can construct accurate metric maps through autonomous exploration. We introduce SpatialScout, a benchmark for evaluating vision-language model (VLM) agents on active object-level 3D mapping. SpatialScout comprises 80 diverse indoor scenes containing 3,140 target objects. Given an object inventory, RGB observations, camera calibration, and camera poses, agents control a camera to explore each scene and estimate object centers in a shared coordinate frame. The task requires coordinating viewpoint selection, cross-view association, geometric estimation, and map revision under bounded interaction budgets. Evaluation jointly measures localization accuracy and target coverage through object-scale overlap scores that penalize missing predictions. Experiments with frontier VLMs reveal substantial limitations: the strongest evaluated model, Claude Opus 5.5, achieves a mean overlap score of 41.71%, with 47.23% of targets exceeding an intersection-over-union (IoU) threshold of 0.5 and only 3.25% exceeding 0.9. These results reveal a substantial gap between recovering coarse object layouts and constructing complete, precise metric maps. SpatialScout provides a quantitative testbed for developing agents that build reliable 3D spatial representations through active perception. Code is available at https://github.com/JesseZZZZZ/SpatialScout.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑