思考、行动、构建：一种基于视觉语言模型的代理框架用于零样本3D视觉定位
Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding
AI总结：
本文提出TAB框架，通过2D到3D重建范式直接处理原始RGB-D流，利用2D VLMs解析复杂空间语义并结合多视图几何构建3D结构，克服传统方法依赖预处理点云的局限，实验证明其在ScanRefer和Nr3D上优于零样本方法和全监督基线。
摘要：
3D视觉定位（3D-VG）旨在通过自然语言描述在3D场景中定位物体。尽管最近利用视觉语言模型（VLMs）的研究探索了零样本可能性，但通常依赖预处理的3D点云，导致定位退化为提案匹配。为避免这种依赖，我们的核心动机是将任务解耦：利用2D VLMs解析复杂空间语义，同时依赖确定性多视图几何构建3D结构。受此启发，我们提出“思考、行动、构建”（TAB）动态代理框架，将3D-VG任务重新表述为直接在原始RGB-D流上进行的生成式2D到3D重建范式。具体而言，受专用3D-VG技能引导，VLM代理动态调用视觉工具在2D帧中跟踪和重建目标。关键的是，为克服严格VLM语义跟踪导致的多视图覆盖不足，我们引入语义锚定几何扩展机制，首先在参考视频片段中锚定目标，然后利用多视图几何将其空间位置传播到未观察帧。这使代理能通过相机参数聚合这些多视图特征，直接将2D视觉线索映射到3D坐标。此外，为确保严格评估，我们识别了现有基准中的缺陷，如参考模糊和类别错误，并手动修正错误查询。在ScanRefer和Nr3D上的大量实验表明，我们的框架完全依赖开源模型，显著优于之前的零样本方法，甚至超越了全监督基线。
英文摘要：
3D Visual Grounding (3D-VG) aims to localize objects in 3D scenes via natural language descriptions. While recent advancements leveraging Vision-Language Models (VLMs) have explored zero-shot possibilities, they typically suffer from a static workflow relying on preprocessed 3D point clouds, essentially degrading grounding into proposal matching. To bypass this reliance, our core motivation is to decouple the task: leveraging 2D VLMs to resolve complex spatial semantics, while relying on deterministic multi-view geometry to instantiate the 3D structure. Driven by this insight, we propose "Think, Act, Build (TAB)", a dynamic agentic framework that reformulates 3D-VG tasks as a generative 2D-to-3D reconstruction paradigm operating directly on raw RGB-D streams. Specifically, guided by a specialized 3D-VG skill, our VLM agent dynamically invokes visual tools to track and reconstruct the target across 2D frames. Crucially, to overcome the multi-view coverage deficit caused by strict VLM semantic tracking, we introduce the Semantic-Anchored Geometric Expansion, a mechanism that first anchors the target in a reference video clip and then leverages multi-view geometry to propagate its spatial location across unobserved frames. This enables the agent to "Build" the target's 3D representation by aggregating these multi-view features via camera parameters, directly mapping 2D visual cues to 3D coordinates. Furthermore, to ensure rigorous assessment, we identify flaws such as reference ambiguity and category errors in existing benchmarks and manually refine the incorrect queries. Extensive experiments on ScanRefer and Nr3D demonstrate that our framework, relying entirely on open-source models, significantly outperforms previous zero-shot methods and even surpasses fully supervised baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
