3D场景图引导的视觉-语言预训练
3D Scene Graph Guided Vision-Language Pre-training
AI总结：
针对现有3D视觉-语言推理方法任务特定、依赖手工模块的局限，提出3D场景图引导的视觉-语言预训练框架，通过模态编码器等学习通用表示，在3D视觉定位等任务上性能优异。
摘要：
3D视觉-语言（VL）推理因其在连接3D物理世界与自然语言描述方面的潜力而备受关注。现有方法通常遵循特定任务、高度专业化的范式，因此这些方法仅聚焦有限范围的推理子任务，并严重依赖手工模块和辅助损失。这凸显了对更简单、统一且通用模型的需求。在本文中，我们利用3D场景图与自然语言之间的内在联系，提出一种3D场景图引导的视觉-语言预训练（VLP）框架。我们的方法利用模态编码器、图卷积层和交叉注意力层来学习适应多种3D VL推理任务的通用表示，从而无需特定任务设计。预训练目标包括：1）场景图引导的对比学习，利用3D场景图与自然语言之间的强相关性，在各种细粒度层面上将3D对象与文本特征对齐；2）掩码模态学习，使用跨模态信息重建掩码词和3D对象。我们不直接重建掩码对象的3D点云，而是使用位置线索预测其语义类别。大量实验表明，我们的预训练模型在多个下游任务（如3D视觉定位、3D密集 captioning 和3D问答）上微调后，性能达到或优于现有方法。
英文摘要：
3D vision-language (VL) reasoning has gained significant attention due to its potential to bridge the 3D physical world with natural language descriptions. Existing approaches typically follow task-specific, highly specialized paradigms. Therefore, these methods focus on a limited range of reasoning sub-tasks and rely heavily on the hand-crafted modules and auxiliary losses. This highlights the need for a simpler, unified and general-purpose model. In this paper, we leverage the inherent connection between 3D scene graphs and natural language, proposing a 3D scene graph-guided vision-language pre-training (VLP) framework. Our approach utilizes modality encoders, graph convolutional layers and cross-attention layers to learn universal representations that adapt to a variety of 3D VL reasoning tasks, thereby eliminating the need for task-specific designs. The pre-training objectives include: 1) Scene graph-guided contrastive learning, which leverages the strong correlation between 3D scene graphs and natural language to align 3D objects with textual features at various fine-grained levels; and 2) Masked modality learning, which uses cross-modality information to reconstruct masked words and 3D objects. Instead of directly reconstructing the 3D point clouds of masked objects, we use position clues to predict their semantic categories. Extensive experiments demonstrate that our pre-training model, when fine-tuned on several downstream tasks, achieves performance comparable to or better than existing methods in tasks such as 3D visual grounding, 3D dense captioning, and 3D question answering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
