SceneVerse：面向场景理解落地的 3D 视觉-语言学习扩展
SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding
AI总结：
针对3D视觉-语言定位中场景复杂、数据稀缺和缺乏统一框架的挑战，提出百万级数据集SceneVerse及统一预训练框架GPS，在多项3D视觉定位基准上达到SOTA并展现零样本迁移潜力。
摘要：
3D视觉-语言定位专注于将语言与3D物理环境对齐，是具身智能体发展的基石。与近期2D领域的进展相比，在3D场景中定位语言面临几个重大挑战：(i) 3D场景固有的复杂性，源于多样的物体配置、丰富的属性及复杂的关系；(ii) 支持定位学习的成对3D视觉-语言数据稀缺；(iii) 缺乏从3D定位数据中提炼知识的统一学习框架。在本工作中，我们旨在通过探究在室内环境中系统性扩展3D视觉-语言学习的潜力，来解决3D视觉-语言的这三个主要挑战。我们引入了首个百万规模的3D视觉-语言数据集SceneVerse，包含约68K个3D室内场景，并由人类标注和我们可扩展的基于场景图的生成方法构建了2.5M视觉-语言对。我们证明这种扩展性使得用于3D视觉-语言学习的统一预训练框架Grounded Pre-training for Scenes (GPS) 成为可能。通过大量实验，我们通过在所有现有3D视觉定位基准上取得最先进性能，展示了GPS的有效性。SceneVerse和GPS的巨大潜力通过在具有挑战性的3D视觉-语言任务中的零样本迁移实验得以展现。项目网站：https://scene-verse.github.io。
英文摘要：
3D vision-language grounding, which focuses on aligning language with the 3D physical environment, stands as a cornerstone in the development of embodied agents. In comparison to recent advancements in the 2D domain, grounding language in 3D scenes faces several significant challenges: (i) the inherent complexity of 3D scenes due to the diverse object configurations, their rich attributes, and intricate relationships; (ii) the scarcity of paired 3D vision-language data to support grounded learning; and (iii) the absence of a unified learning framework to distill knowledge from grounded 3D data. In this work, we aim to address these three major challenges in 3D vision-language by examining the potential of systematically upscaling 3D vision-language learning in indoor environments. We introduce the first million-scale 3D vision-language dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising 2.5M vision-language pairs derived from both human annotations and our scalable scene-graph-based generation approach. We demonstrate that this scaling allows for a unified pre-training framework, Grounded Pre-training for Scenes (GPS), for 3D vision-language learning. Through extensive experiments, we showcase the effectiveness of GPS by achieving state-of-the-art performance on all existing 3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is unveiled through zero-shot transfer experiments in the challenging 3D vision-language tasks. Project website: https://scene-verse.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
