arXiv 2503.232972025-03-30cs.CV

ReasonGrounder：用于开放词汇3D视觉定位与推理的LVLM引导分层特征泼溅

ReasonGrounder: LVLM-Guided Hierarchical Feature Splatting for Open-Vocabulary 3D Visual Grounding and Reasoning

Fudan University（复旦大学）；Nanyang Technological University（南洋理工大学）；Shanghai Innovation Institute（上海创新研究院）；NeuHelium Co., Ltd（NeuHelium公司）

Zhenyang Liu, Yikai Wang, Sixiao Zheng, Tongying Pan, Longfei Liang, Yanwei Fu, Xiangyang Xue

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出LVLM引导的框架ReasonGrounder，利用分层3D特征高斯场实现开放词汇3D定位与推理，结合SAM和CLIP处理遮挡，并贡献新数据集ReasoningGD，显著提升真实场景定位准确率。

摘要：

开放词汇3D视觉定位与推理旨在根据隐式语言描述定位场景中的物体，即使它们被遮挡。这种能力对于视觉语言导航和自主机器人等任务至关重要。然而，当前方法存在困难，因为它们严重依赖带有3D标注和掩码提议的微调，这限制了它们处理有效推理所需的多样语义和常识的能力。在这项工作中，我们提出了ReasonGrounder，这是一个LVLM引导的框架，它使用分层3D特征高斯场基于物理尺度进行自适应分组，从而实现开放词汇3D定位与推理。ReasonGrounder使用大型视觉语言模型（LVLM）解释隐式指令，并通过3D高斯泼溅定位被遮挡的物体。通过结合来自SAM的2D分割掩码和多视图CLIP嵌入，ReasonGrounder基于物体尺度选择高斯组，即使在新颖的、被遮挡的视图中，也能通过显式和隐式语言理解实现准确定位。我们还贡献了ReasoningGD，这是一个新数据集，包含超过10K个场景和200万个标注，用于评估遮挡条件下的开放词汇3D定位和模态感知。实验表明，ReasonGrounder显著提高了真实场景中的3D定位准确率。

英文摘要：

Open-vocabulary 3D visual grounding and reasoning aim to localize objects in a scene based on implicit language descriptions, even when they are occluded. This ability is crucial for tasks such as vision-language navigation and autonomous robotics. However, current methods struggle because they rely heavily on fine-tuning with 3D annotations and mask proposals, which limits their ability to handle diverse semantics and common knowledge required for effective reasoning. In this work, we propose ReasonGrounder, an LVLM-guided framework that uses hierarchical 3D feature Gaussian fields for adaptive grouping based on physical scale, enabling open-vocabulary 3D grounding and reasoning. ReasonGrounder interprets implicit instructions using large vision-language models (LVLM) and localizes occluded objects through 3D Gaussian splatting. By incorporating 2D segmentation masks from the SAM and multi-view CLIP embeddings, ReasonGrounder selects Gaussian groups based on object scale, enabling accurate localization through both explicit and implicit language understanding, even in novel, occluded views. We also contribute ReasoningGD, a new dataset containing over 10K scenes and 2 million annotations for evaluating open-vocabulary 3D grounding and amodal perception under occlusion. Experiments show that ReasonGrounder significantly improves 3D grounding accuracy in real-world scenarios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。