arXiv 2411.145942025-08-11cs.CV

将零样本3D视觉定位求解为约束满足问题

Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problems

Bernoulli Institute for Mathematics, Computer Science and Artificial Intelligence, University of Groningen（格罗宁根大学伯努利学院）；Computer Vision for Human-Computer Interaction Lab (cv:hci), Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院人机交互计算机视觉实验室）

Qihao Yuan, Kailai Li, Jiaming Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出将零样本3D视觉定位重新表述为约束满足问题（CSP）的方法CSVG，通过全局符号推理物体空间关系，在ScanRefer和Nr3D数据集上分别提升+7.0%和+11.2%准确率。

摘要：

3D视觉定位（3DVG）旨在通过自然语言描述在3D场景中定位物体。有监督方法取得了不错的准确率，但词汇表封闭且语言理解能力有限。零样本方法利用大型语言模型（LLMs）处理自然语言描述，其中LLM要么直接产生定位结果，要么生成计算结果的程序（符号化）。在本研究中，我们提出一种零样本方法，将3DVG任务重新表述为约束满足问题（CSP），其中变量和约束分别表示物体及其空间关系。这允许对所有相关物体进行全局符号推理，产生目标物体和锚点物体的定位结果。此外，我们展示了框架的灵活性，仅需少量额外编码即可处理否定和基于计数的查询。我们的系统约束满足视觉定位（CSVG）已在公开数据集ScanRefer和Nr3D上进行了广泛评估，仅使用开源LLM。结果显示了CSVG的有效性，在ScanRefer和Nr3D数据集上分别取得了+7.0%（Acc@0.5分数）和+11.2%的提升，优于当前最先进的零样本3DVG方法。系统代码可在https://asig-x.github.io/csvg_web获取。

英文摘要：

3D visual grounding (3DVG) aims to locate objects in a 3D scene with natural language descriptions. Supervised methods have achieved decent accuracy, but have a closed vocabulary and limited language understanding ability. Zero-shot methods utilize large language models (LLMs) to handle natural language descriptions, where the LLM either produces grounding results directly or generates programs that compute results (symbolically). In this work, we propose a zero-shot method that reformulates the 3DVG task as a Constraint Satisfaction Problem (CSP), where the variables and constraints represent objects and their spatial relations, respectively. This allows a global symbolic reasoning of all relevant objects, producing grounding results of both the target and anchor objects. Moreover, we demonstrate the flexibility of our framework by handling negation- and counting-based queries with only minor extra coding efforts. Our system, Constraint Satisfaction Visual Grounding (CSVG), has been extensively evaluated on the public datasets ScanRefer and Nr3D datasets using only open-source LLMs. Results show the effectiveness of CSVG and superior grounding accuracy over current state-of-the-art zero-shot 3DVG methods with improvements of $+7.0\%$ (Acc@0.5 score) and $+11.2\%$ on the ScanRefer and Nr3D datasets, respectively. The code of our system is available at https://asig-x.github.io/csvg_web.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。