arXiv 2308.043522023-08-08cs.CV

3D-VisTA：用于三维视觉与文本对齐的预训练Transformer

3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment

Tsinghua University（清华大学）；National Key Laboratory of General Artificial Intelligence, BIGAI（北京通用人工智能研究院通用人工智能国家重点实验室）

Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, Qing Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出3D-VisTA，一个仅用自注意力层实现三维视觉与文本对齐的预训练Transformer，并构建大规模数据集ScanScribe进行预训练，在多种3D-VL任务上取得最优结果且数据效率高。

摘要：

三维视觉-语言 grounding（3D-VL）是一个新兴领域，旨在将三维物理世界与自然语言连接起来，这对于实现具身智能至关重要。当前的3D-VL模型严重依赖复杂的模块、辅助损失和优化技巧，这呼唤一个简单且统一的模型。在本文中，我们提出了3D-VisTA，一个用于三维视觉与文本对齐的预训练Transformer，它可以轻松适配各种下游任务。3D-VisTA仅利用自注意力层进行单模态建模和多模态融合，没有任何复杂的任务特定设计。为了进一步提升其在3D-VL任务上的性能，我们构建了ScanScribe，这是第一个用于3D-VL预训练的大规模三维场景-文本对数据集。ScanScribe包含2,995个RGB-D扫描，涵盖来自ScanNet和3R-Scan数据集的1,185个独特室内场景，以及配对的278K条场景描述，这些描述由现有的3D-VL任务、模板和GPT-3生成。3D-VisTA通过掩码语言/物体建模和场景-文本匹配在ScanScribe上进行预训练。它在各种3D-VL任务上取得了最先进的结果，涵盖视觉grounding、密集描述、问答和情境推理。此外，3D-VisTA展示了卓越的数据效率，即使在下游任务微调期间只有有限的标注，也能获得强大的性能。

英文摘要：

3D vision-language grounding (3D-VL) is an emerging field that aims to connect the 3D physical world with natural language, which is crucial for achieving embodied intelligence. Current 3D-VL models rely heavily on sophisticated modules, auxiliary losses, and optimization tricks, which calls for a simple and unified model. In this paper, we propose 3D-VisTA, a pre-trained Transformer for 3D Vision and Text Alignment that can be easily adapted to various downstream tasks. 3D-VisTA simply utilizes self-attention layers for both single-modal modeling and multi-modal fusion without any sophisticated task-specific design. To further enhance its performance on 3D-VL tasks, we construct ScanScribe, the first large-scale 3D scene-text pairs dataset for 3D-VL pre-training. ScanScribe contains 2,995 RGB-D scans for 1,185 unique indoor scenes originating from ScanNet and 3R-Scan datasets, along with paired 278K scene descriptions generated from existing 3D-VL tasks, templates, and GPT-3. 3D-VisTA is pre-trained on ScanScribe via masked language/object modeling and scene-text matching. It achieves state-of-the-art results on various 3D-VL tasks, ranging from visual grounding and dense captioning to question answering and situated reasoning. Moreover, 3D-VisTA demonstrates superior data efficiency, obtaining strong performance even with limited annotations during downstream task fine-tuning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。