AgentGrounder：使用多模态语言模型的零样本3D视觉点云定位
AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models
AI总结：
提出AgentGrounder框架，通过两阶段设计（离线构建对象查找表和在线工具驱动代理）实现零样本3D视觉定位，在ScanRefer和Nr3D上分别提升2.5%和6.3%的准确率。
摘要：
3D视觉定位（3DVG）是具身AI的基本能力，要求智能体根据自然语言描述在3D场景中定位物体。最近的零样本方法利用2D视觉语言模型（LVLMs），但它们通常依赖于现有的多视图图像集，并且难以处理标准3D分割工具提供的有限语义和空间细节。我们提出了$ extbf{AgentGrounder}$，一个零样本3D视觉定位框架，直接对彩色点云进行操作，无需特定任务的3D训练。我们的方法采用两阶段设计：（1）离线阶段，应用3D模型构建对象查找表（OLT），包含实例ID、语义标签、3D边界框；（2）在线工具驱动代理，分解每个查询，仅从OLT中检索相关候选对象，进行几何评分，并在需要额外视觉证据（如颜色、材质或视角敏感线索）时按需触发图像渲染。与固定的锚点-目标匹配流水线相比，这种设计减少了级联匹配错误，并通过避免提示过载无关对象来提高上下文窗口效率。我们在零样本设置下对ScanRefer和Nr3D进行了评估，观察到在我们的设置中比SeeGround有持续改进，包括ScanRefer上+2.5%的Acc@0.5和Nr3D上+6.3%，在Nr3D视图无关查询上显著提升+6.3%。这些结果表明，结合选择性检索、几何推理和自适应视觉检查为开放词汇3D定位提供了实用且稳健的基础。我们的代码可在https://github.com/be2rlab/AgentGrounder获取。
英文摘要：
3D Visual Grounding (3DVG) is an essential capability for embodied AI, requiring agents to localize objects in 3D scenes based on natural language descriptions. Recent zero-shot methods leverage 2D vision-language models (LVLMs). However, they often rely on existing sets of multi-view images and struggle with the limited semantic and spatial details provided by standard 3D segmentation tools. We present $\textbf{AgentGrounder}$, a zero-shot 3D visual grounding framework that operates directly on colored point clouds without task-specific 3D training. Our approach follows a two-stage design: (1) an offline stage that applies 3D model to build an Object Lookup Table (OLT) with instance IDs, semantic labels, 3D bounding boxes; and (2) an online tool-driven agent that decomposes each query, retrieves only relevant candidates from the OLT, performs geometric scoring, and triggers image rendering on demand when additional visual evidence (e.g., color, material, or viewpoint-sensitive cues) is required. Compared with fixed anchor-target matching pipelines, this design reduces cascading matching errors and improves context-window efficiency by avoiding prompts overloaded with irrelevant objects. We evaluate on ScanRefer and Nr3D under a zero-shot setting and observe consistent improvements over SeeGround in our setup, including +2.5% Acc@0.5 on ScanRefer and +6.3% on Nr3D, with a notable +6.3% gain on Nr3D view-independent queries. These results show that combining selective retrieval, geometric reasoning, and adaptive visual inspection yields a practical and robust foundation for open-vocabulary 3D grounding. Our code is available at https://github.com/be2rlab/AgentGrounder.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
