VLM-Grounder：一种用于零样本3D视觉定位的VLM智能体
VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding
AI总结：
本文提出VLM-Grounder，一种仅用2D图像和视觉语言模型实现零样本3D视觉定位的框架，通过动态图像拼接、定位反馈和多视角集成投影，在ScanRefer和Nr3D上超越已有零样本方法。
摘要：
3D视觉定位对机器人至关重要，需要融合自然语言与3D场景理解。依赖3D点云监督学习的传统方法受限于数据集稀缺。近期，利用大语言模型（LLMs）的零样本方法被提出以解决数据问题。这些方法虽然有效，但仅使用以物体为中心的信息，限制了其处理复杂查询的能力。在本工作中，我们提出VLM-Grounder，一种仅基于2D图像、利用视觉语言模型（VLMs）进行零样本3D视觉定位的新框架。VLM-Grounder动态拼接图像序列，采用定位与反馈机制来寻找目标物体，并使用多视角集成投影来准确估计3D边界框。在ScanRefer和Nr3D数据集上的实验表明，VLM-Grounder优于以往的零样本方法，在ScanRefer上达到51.6%的Acc@0.25，在Nr3D上达到48.0%的Acc，且不依赖3D几何信息或物体先验。代码可在https://github.com/OpenRobotLab/VLM-Grounder获取。
英文摘要：
3D visual grounding is crucial for robots, requiring integration of natural language and 3D scene understanding. Traditional methods depending on supervised learning with 3D point clouds are limited by scarce datasets. Recently zero-shot methods leveraging LLMs have been proposed to address the data issue. While effective, these methods only use object-centric information, limiting their ability to handle complex queries. In this work, we present VLM-Grounder, a novel framework using vision-language models (VLMs) for zero-shot 3D visual grounding based solely on 2D images. VLM-Grounder dynamically stitches image sequences, employs a grounding and feedback scheme to find the target object, and uses a multi-view ensemble projection to accurately estimate 3D bounding boxes. Experiments on ScanRefer and Nr3D datasets show VLM-Grounder outperforms previous zero-shot methods, achieving 51.6% Acc@0.25 on ScanRefer and 48.0% Acc on Nr3D, without relying on 3D geometry or object priors. Codes are available at https://github.com/OpenRobotLab/VLM-Grounder .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
