SeeGround：面向零样本开放词汇3D视觉定位的观察与定位
SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding
AI总结：
本文提出零样本3D视觉定位框架SeeGround，利用2D视觉语言模型，通过视角自适应和融合对齐模块结合渲染图像与空间文本，在ScanRefer和Nr3D上大幅超越现有零样本方法。
摘要：
3D视觉定位（3DVG）旨在根据文本描述在3D场景中定位物体，这对于增强现实和机器人等应用至关重要。传统的3DVG方法依赖于带标注的3D数据集和预定义的物体类别，限制了可扩展性和适应性。为了克服这些局限性，我们提出了SeeGround，这是一个利用在大规模2D数据上训练的2D视觉语言模型（VLMs）的零样本3DVG框架。SeeGround将3D场景表示为查询对齐的渲染图像和空间增强的文本描述的混合体，弥合了3D数据与2D-VLMs输入格式之间的差距。我们提出了两个模块：视角自适应模块，动态选择视点以渲染与查询相关的图像；融合对齐模块，将2D图像与3D空间描述相结合以增强物体定位。在ScanRefer和Nr3D上的大量实验表明，我们的方法大幅超越了现有的零样本方法。值得注意的是，我们超越了弱监督方法，并可与一些全监督方法相媲美，在ScanRefer上比之前的SOTA高出7.7%，在Nr3D上高出7.1%，展示了其在复杂3DVG任务中的有效性。
英文摘要：
3D Visual Grounding (3DVG) aims to locate objects in 3D scenes based on textual descriptions, essential for applications like augmented reality and robotics. Traditional 3DVG approaches rely on annotated 3D datasets and predefined object categories, limiting scalability and adaptability. To overcome these limitations, we introduce SeeGround, a zero-shot 3DVG framework leveraging 2D Vision-Language Models (VLMs) trained on large-scale 2D data. SeeGround represents 3D scenes as a hybrid of query-aligned rendered images and spatially enriched text descriptions, bridging the gap between 3D data and 2D-VLMs input formats. We propose two modules: the Perspective Adaptation Module, which dynamically selects viewpoints for query-relevant image rendering, and the Fusion Alignment Module, which integrates 2D images with 3D spatial descriptions to enhance object localization. Extensive experiments on ScanRefer and Nr3D demonstrate that our approach outperforms existing zero-shot methods by large margins. Notably, we exceed weakly supervised methods and rival some fully supervised ones, outperforming previous SOTA by 7.7% on ScanRefer and 7.1% on Nr3D, showcasing its effectiveness in complex 3DVG tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
