DenseGrounding：提升以自我为中心的三维视觉定位中的密集语言-视觉语义
DenseGrounding: Improving Dense Language-Vision Semantics for Ego-Centric 3D Visual Grounding
AI总结：
DenseGrounding针对以自我为中心的三维视觉定位中视觉语义稀疏和文本上下文有限的问题，提出层次化场景语义增强器和基于大语言模型的语言语义增强器，显著提升定位准确率并在CVPR 2024挑战赛中获得第一名。
摘要：
使智能体能够通过自然语言理解并与三维环境交互，对于推动机器人技术和人机交互的发展至关重要。该领域的一项基础任务是以自我为中心的三维视觉定位，即智能体根据语言描述在真实三维空间中定位目标物体。然而，该任务面临两个重大挑战：（1）由于点云与以自我为中心的多视角图像融合稀疏，导致细粒度视觉语义丢失；（2）由于语言描述具有任意性，导致文本语义上下文有限。我们提出DenseGrounding，这是一种旨在通过同时增强视觉和文本语义来解决上述问题的新方法。在视觉特征方面，我们引入了层次化场景语义增强器（Hierarchical Scene Semantic Enhancer），通过捕获细粒度的全局场景特征并促进跨模态对齐来保留密集语义。在文本描述方面，我们提出了语言语义增强器（Language Semantic Enhancer），利用大语言模型在模型训练期间提供丰富的上下文以及具有额外上下文的多样化语言描述。大量实验表明，DenseGrounding在整体准确率上显著优于现有方法，在完整数据集和较小的mini子集上训练时分别提升了5.81%和7.56%，进一步推进了以自我为中心的三维视觉定位的最先进水平（SOTA）。我们的方法还在CVPR 2024 Autonomous Grand Challenge多视角三维视觉定位赛道中获得第一名并荣获创新奖，验证了其有效性和鲁棒性。
英文摘要：
Enabling intelligent agents to comprehend and interact with 3D environments through natural language is crucial for advancing robotics and human-computer interaction. A fundamental task in this field is ego-centric 3D visual grounding, where agents locate target objects in real-world 3D spaces based on verbal descriptions. However, this task faces two significant challenges: (1) loss of fine-grained visual semantics due to sparse fusion of point clouds with ego-centric multi-view images, (2) limited textual semantic context due to arbitrary language descriptions. We propose DenseGrounding, a novel approach designed to address these issues by enhancing both visual and textual semantics. For visual features, we introduce the Hierarchical Scene Semantic Enhancer, which retains dense semantics by capturing fine-grained global scene features and facilitating cross-modal alignment. For text descriptions, we propose a Language Semantic Enhancer that leverages large language models to provide rich context and diverse language descriptions with additional context during model training. Extensive experiments show that DenseGrounding significantly outperforms existing methods in overall accuracy, with improvements of 5.81% and 7.56% when trained on the comprehensive full dataset and smaller mini subset, respectively, further advancing the SOTA in egocentric 3D visual grounding. Our method also achieves 1st place and receives the Innovation Award in the CVPR 2024 Autonomous Grand Challenge Multi-view 3D Visual Grounding Track, validating its effectiveness and robustness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
