arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.03026cs.CV
收藏

Uni3DL:用于3D与语言理解的统一模型

Uni3DL: Unified Model for 3D and Language Understanding

King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Ecole Polytechnique(巴黎综合理工学院)

Xiang Li, Jian Ding, Zhaoyang Chen, Mohamed Elhoseiny

AI总结:

本文提出直接处理点云的统一模型Uni3DL,通过query transformer与task router支持多种3D视觉和视觉-语言任务,并在多项基准上达到或超过SOTA。

摘要:

在本工作中,我们提出了Uni3DL,这是一个用于3D与语言理解的统一模型。与现有3D领域统一视觉-语言模型不同,这些模型受限于任务种类,并且主要依赖投影得到的多视图图像;Uni3DL直接在点云上运行。该方法显著扩展了3D中可支持任务的范围,涵盖3D视觉任务和3D视觉-语言任务。Uni3DL的核心是一个query transformer,旨在通过关注3D视觉特征来学习与任务无关的语义和掩码输出;同时采用一个task router,以选择性地生成不同任务所需的任务特定输出。借助统一架构,我们的Uni3DL模型能够实现无缝的任务分解,并在各任务之间进行大量参数共享。Uni3DL已在多种3D视觉-语言理解任务上接受严格评估,包括语义分割、目标检测、实例分割、视觉定位、3D描述生成以及文本-3D跨模态检索。结果表明,其性能与最先进(SOTA)的任务特定模型相当,甚至有所超越。我们希望该基准和Uni3DL模型能够成为坚实的一步,推动未来3D与语言理解领域中统一模型的研究。项目页面:https://uni3dl.github.io。

英文摘要:

In this work, we present Uni3DL, a unified model for 3D and Language understanding. Distinct from existing unified vision-language models in 3D which are limited in task variety and predominantly dependent on projected multi-view images, Uni3DL operates directly on point clouds. This approach significantly expands the range of supported tasks in 3D, encompassing both vision and vision-language tasks in 3D. At the core of Uni3DL, a query transformer is designed to learn task-agnostic semantic and mask outputs by attending to 3D visual features, and a task router is employed to selectively generate task-specific outputs required for diverse tasks. With a unified architecture, our Uni3DL model enjoys seamless task decomposition and substantial parameter sharing across tasks. Uni3DL has been rigorously evaluated across diverse 3D vision-language understanding tasks, including semantic segmentation, object detection, instance segmentation, visual grounding, 3D captioning, and text-3D cross-modal retrieval. It demonstrates performance on par with or surpassing state-of-the-art (SOTA) task-specific models. We hope our benchmark and Uni3DL model will serve as a solid step to ease future research in unified models in the realm of 3D and language understanding. Project page: https://uni3dl.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑