一种具有LLM驱动的空间推理的神经表示框架，用于开放词汇3D视觉定位
A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding
AI总结：
提出SpatialReasoner框架，利用LLM驱动的空间推理和视觉属性增强的分层特征场，解决开放词汇3D视觉定位中空间关系推理不足的问题，显著提升定位性能。
摘要：
开放词汇3D视觉定位旨在基于自由形式的语言查询定位目标对象，这对于具身人工智能应用（如自主导航、机器人和增强现实）至关重要。通过神经表示学习3D语言场能够从有限的视角准确理解3D场景，并有助于在复杂环境中定位目标对象。然而，现有的语言场方法难以利用语言查询中的空间关系（如“椅子上的书”）准确定位实例。这一局限性主要源于对语言查询和3D场景中空间关系的推理不足。在这项工作中，我们提出了SpatialReasoner，一种新颖的基于神经表示的框架，具有大语言模型（LLM）驱动的空间推理，该框架构建了一个视觉属性增强的分层特征场，用于开放词汇3D视觉定位。为了在语言查询中实现空间推理，SpatialReasoner微调了一个LLM以捕获空间关系，并显式推断目标、锚点和空间关系的指令。为了在3D场景中实现空间推理，SpatialReasoner整合了视觉属性（不透明度和颜色）来构建分层特征场。该场使用蒸馏的CLIP特征和通过分段任意模型（SAM）提取的掩码来表示语言和实例特征。然后，使用推断的指令以分层方式查询该场，以基于语言查询中的空间关系定位目标3D实例。大量实验表明，我们的框架可以无缝集成到不同的神经表示中，在3D视觉定位中优于基线模型，同时增强了其空间推理能力。
英文摘要：
Open-vocabulary 3D visual grounding aims to localize target objects based on free-form language queries, which is crucial for embodied AI applications such as autonomous navigation, robotics, and augmented reality. Learning 3D language fields through neural representations enables accurate understanding of 3D scenes from limited viewpoints and facilitates the localization of target objects in complex environments. However, existing language field methods struggle to accurately localize instances using spatial relations in language queries, such as ``the book on the chair.'' This limitation mainly arises from inadequate reasoning about spatial relations in both language queries and 3D scenes. In this work, we propose SpatialReasoner, a novel neural representation-based framework with large language model (LLM)-driven spatial reasoning that constructs a visual properties-enhanced hierarchical feature field for open-vocabulary 3D visual grounding. To enable spatial reasoning in language queries, SpatialReasoner fine-tunes an LLM to capture spatial relations and explicitly infer instructions for the target, anchor, and spatial relation. To enable spatial reasoning in 3D scenes, SpatialReasoner incorporates visual properties (opacity and color) to construct a hierarchical feature field. This field represents language and instance features using distilled CLIP features and masks extracted via the Segment Anything Model (SAM). The field is then queried using the inferred instructions in a hierarchical manner to localize the target 3D instance based on the spatial relation in the language query. Extensive experiments show that our framework can be seamlessly integrated into different neural representations, outperforming baseline models in 3D visual grounding while empowering their spatial reasoning capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
