提出SpatialScout基准评估VLM智能体的主动对象级3D地图绘制能力,实验显示最强模型Claude Opus 5.5仅达41.71%平均重叠分,暴露了构建精确3D地图的显著差距。
摘要:
空间智能要求智能体主动获取视觉证据,并将观测结果整合为三维世界的连贯表示。然而空间问答仅能提供有限证据,无法证明模型是否能通过自主探索构建精确的度量地图。我们提出SpatialScout,这是一个用于评估视觉语言模型(VLM)智能体在主动对象级3D地图绘制任务中的基准。SpatialScout包含80个多样化的室内场景,共含3140个目标对象。给定对象清单、RGB观测值、相机标定参数和相机位姿,智能体控制相机探索每个场景,并在共享坐标系中估计对象中心。该任务要求在有限的交互预算下协调视点选择、跨视图关联、几何估计和地图修正。评估通过对象级重叠分数联合衡量定位精度和目标覆盖率,该分数会惩罚缺失预测。对前沿VLM的实验显示出显著局限性:评估中最强的模型Claude Opus 5.5的平均重叠分数为41.71%,其中47.23%的目标超过交并比(IoU)阈值0.5,仅3.25%超过0.9。这些结果表明,恢复粗略对象布局与构建完整精确的度量地图之间存在显著差距。SpatialScout为开发通过主动感知构建可靠3D空间表示的智能体提供了定量测试平台。代码可在此https URL获取。
英文摘要:
Spatial intelligence requires agents to actively acquire visual evidence and integrate observations into a coherent representation of the three-dimensional world. Yet spatial question answering offers limited evidence of whether models can construct accurate metric maps through autonomous exploration. We introduce SpatialScout, a benchmark for evaluating vision-language model (VLM) agents on active object-level 3D mapping. SpatialScout comprises 80 diverse indoor scenes containing 3,140 target objects. Given an object inventory, RGB observations, camera calibration, and camera poses, agents control a camera to explore each scene and estimate object centers in a shared coordinate frame. The task requires coordinating viewpoint selection, cross-view association, geometric estimation, and map revision under bounded interaction budgets. Evaluation jointly measures localization accuracy and target coverage through object-scale overlap scores that penalize missing predictions. Experiments with frontier VLMs reveal substantial limitations: the strongest evaluated model, Claude Opus 5.5, achieves a mean overlap score of 41.71%, with 47.23% of targets exceeding an intersection-over-union (IoU) threshold of 0.5 and only 3.25% exceeding 0.9. These results reveal a substantial gap between recovering coarse object layouts and constructing complete, precise metric maps. SpatialScout provides a quantitative testbed for developing agents that build reliable 3D spatial representations through active perception. Code is available at https://github.com/JesseZZZZZ/SpatialScout.
Immuno-VLM: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness
School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院);Nanyang Technological University, Singapore(新加坡南洋理工大学);Nanjing University(南京大学)
University of Maryland, College Park(马里兰大学帕克分校);University of California, Los Angeles(加州大学洛杉矶分校);Peking University(北京大学);Arena;MBZUAI(穆罕默德·本·扎耶德人工智能大学)
Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators
The University of Hong Kong(香港大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学);Beihang University(北航大学)
Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction
AIML Lab, TU Darmstadt(人工智能机器学习实验室,达姆施塔特工业大学);Hessian Center for AI (hessian.AI)(黑森州人工智能中心);Psychology of Information Processing, TU Darmstadt(信息处理心理学,达姆施塔特工业大学);Centre for Cognitive Science, TU Darmstadt(认知科学中心,达姆施塔特工业大学);Cornell University(康奈尔大学);Max Planck Institute for Informatics, SIC(马克斯·普朗克信息学研究所,SIC组);German Center for AI (DFKI)(德国人工智能研究中心)
Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning
University of California, Berkeley(加州大学伯克利分校);Carnegie Mellon University(卡内基梅隆大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Impossible, Inc.(Impossible公司)
Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China(华中科技大学计算机科学与技术学院);Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China(华中科技大学人工智能研究院)
Agent3D-Zero: An Agent for Zero-shot 3D Understanding
University of Science and Technology of China(中国科学技术大学);Shanghai Artificial Intelligent Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);The University of Adelaide(阿德莱德大学);Australian Institute for Machine Learning(澳大利亚机器学习研究所)
E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes
The University of Tokyo, Japan(东京大学);National Institute of Informatics, Japan(日本信息处理学会);The University of Osaka, Japan(大阪大学);Advanced Telecommunications Research Institute International, Japan(国际先进电信研究机构)