面向零样本开放词汇3D视觉定位的视觉编程
Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding
AI总结：
提出基于大语言模型的零样本开放词汇3D视觉定位视觉编程方法，通过三类协同模块和创新语言-对象关联模块，无需标注即超越部分监督基线。
摘要：
3D视觉定位（3DVG）旨在根据文本描述定位3D对象。传统的3DVG监督方法通常需要大量标注和预定义词汇表，这存在局限性。为解决此问题，我们提出了一种新颖的零样本开放词汇3DVG视觉编程方法，利用大语言模型（LLMs）的能力。该方法首先采用独特的基于对话的方法，与LLMs交互以建立对零样本3DVG的基础理解。在此基础上，我们设计了一个由三类模块组成的视觉程序，即视图无关模块、视图相关模块和功能模块。这些专为3D场景定制的模块协同工作，执行复杂推理与推断。此外，我们开发了一个创新的语言-对象关联模块，将现有3D对象检测器的范围扩展到开放词汇场景。大量实验表明，我们的零样本方法能超越部分监督基线，标志着向有效3DVG迈出的重要一步。
英文摘要：
3D Visual Grounding (3DVG) aims at localizing 3D object based on textual descriptions. Conventional supervised methods for 3DVG often necessitate extensive annotations and a predefined vocabulary, which can be restrictive. To address this issue, we propose a novel visual programming approach for zero-shot open-vocabulary 3DVG, leveraging the capabilities of large language models (LLMs). Our approach begins with a unique dialog-based method, engaging with LLMs to establish a foundational understanding of zero-shot 3DVG. Building on this, we design a visual program that consists of three types of modules, i.e., view-independent, view-dependent, and functional modules. These modules, specifically tailored for 3D scenarios, work collaboratively to perform complex reasoning and inference. Furthermore, we develop an innovative language-object correlation module to extend the scope of existing 3D object detectors into open-vocabulary scenarios. Extensive experiments demonstrate that our zero-shot approach can outperform some supervised baselines, marking a significant stride towards effective 3DVG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
