抽象3D感知用于视觉-语言模型中的空间智能
Abstract 3D Perception for Spatial Intelligence in Vision-Language Models
AI总结：
本文提出SandboxVLM框架，通过抽象边界框编码几何结构和物理动力学，提升视觉-语言模型在3D任务中的空间智能，实验证明其在零样本设置下显著提升性能。
摘要：
视觉-语言模型（VLMs）在涉及空间认知和物理理解的3D任务中表现不佳，这对机器人和具身智能等实际应用至关重要。我们归因于VLMs的2D训练与3D任务之间的模态差距，导致从2D输入中检索3D信息效率低下。为弥合这一差距，我们引入SandboxVLM，一个简单而有效的框架，利用抽象边界框编码几何结构和物理动力学。具体而言，我们设计了一个包含四个阶段的3D Sandbox重建和感知流水线：生成多视角先验、代理海拔、多视角投票和聚类，以及3D-aware推理。在多个基准和VLM后端的零样本设置下评估，我们的方法一致提升了空间智能，实例在SAT Real上比基线方法提高了8.3%。这些结果表明，为VLMs提供3D抽象能显著增强其3D推理能力，而无需额外训练，暗示了通用具身智能的新可能性。
英文摘要：
Vision-language models (VLMs) struggle with 3D-related tasks such as spatial cognition and physical understanding, which are crucial for real-world applications like robotics and embodied agents. We attribute this to a modality gap between the 3D tasks and the 2D training of VLM, which led to inefficient retrieval of 3D information from 2D input. To bridge this gap, we introduce SandboxVLM, a simple yet effective framework that leverages abstract bounding boxes to encode geometric structure and physical kinematics for VLM. Specifically, we design a 3D Sandbox reconstruction and perception pipeline comprising four stages: generating multi-view priors with abstract control, proxy elevation, multi-view voting and clustering, and 3D-aware reasoning. Evaluated in zero-shot settings across multiple benchmarks and VLM backbones, our approach consistently improves spatial intelligence, achieving an 8.3\% gain on SAT Real compared with baseline methods for instance. These results demonstrate that equipping VLMs with a 3D abstraction substantially enhances their 3D reasoning ability without additional training, suggesting new possibilities for general-purpose embodied intelligence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
