arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.11835cs.CV
收藏

Agent3D-Zero:用于零样本3D理解的智能体

Agent3D-Zero: An Agent for Zero-shot 3D Understanding

University of Science and Technology of China(中国科学技术大学);Shanghai Artificial Intelligent Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);The University of Adelaide(阿德莱德大学);Australian Institute for Machine Learning(澳大利亚机器学习研究所)

Sha Zhang, Di Huang, Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, Yanyong Zhang

AI总结:

Agent3D-Zero提出零样本3D理解智能体框架,通过视觉提示引导VLM主动选择并分析多视角图像,实现对未见3D场景的有效理解。

摘要:

理解和推理3D真实世界的能力是迈向通用人工智能的关键里程碑。目前常见的做法是利用3D数据和文本对大型语言模型(LLMs)进行微调,以实现3D理解。尽管这些方法有效,但它们本质上受到可用3D数据的规模和多样性的限制。在本工作中,我们另辟蹊径,引入了Agent3D-Zero,一个创新的3D感知智能体框架,以零样本的方式解决3D场景理解问题。我们方法的核心在于将3D场景感知的挑战重新构想为一个从多张图像中理解和综合洞察的过程,这受到人类试图理解3D场景方式的启发。通过整合这一思想,我们提出了一种新颖的方式,利用大型视觉语言模型(VLM),通过主动选择和分析一系列视角来进行3D理解。具体来说,给定一个输入的3D场景,Agent3D-Zero首先使用定制设计的视觉提示处理鸟瞰图,然后迭代地选择下一个要观察的视角并总结底层知识。Agent3D-Zero的一个显著优势是引入了新颖的视觉提示,这极大地释放了VLM识别最具信息量视角的能力,从而促进了对3D场景的观察。大量实验证明了该框架在理解多样且未见过的3D环境方面的有效性。

英文摘要:

The ability to understand and reason the 3D real world is a crucial milestone towards artificial general intelligence. The current common practice is to finetune Large Language Models (LLMs) with 3D data and texts to enable 3D understanding. Despite their effectiveness, these approaches are inherently limited by the scale and diversity of the available 3D data. Alternatively, in this work, we introduce Agent3D-Zero, an innovative 3D-aware agent framework addressing the 3D scene understanding in a zero-shot manner. The essence of our approach centers on reconceptualizing the challenge of 3D scene perception as a process of understanding and synthesizing insights from multiple images, inspired by how our human beings attempt to understand 3D scenes. By consolidating this idea, we propose a novel way to make use of a Large Visual Language Model (VLM) via actively selecting and analyzing a series of viewpoints for 3D understanding. Specifically, given an input 3D scene, Agent3D-Zero first processes a bird's-eye view image with custom-designed visual prompts, then iteratively chooses the next viewpoints to observe and summarize the underlying knowledge. A distinctive advantage of Agent3D-Zero is the introduction of novel visual prompts, which significantly unleash the VLMs' ability to identify the most informative viewpoints and thus facilitate observing 3D scenes. Extensive experiments demonstrate the effectiveness of the proposed framework in understanding diverse and previously unseen 3D environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑