arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.03540cs.RO
收藏

VLA-3D:用于3D语义场景理解与导航的数据集

VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation

Robotics Institute, Carnegie Mellon University(卡内基梅隆大学机器人研究所)

Haochen Zhang, Nader Zantout, Pujith Kachana, Zongyuan Wu, Ji Zhang, Wenshan Wang

AI总结:

为应对室内导航中的空间推理与语义理解挑战,构建了最大真实世界数据集VLA-3D,含11.5K房间、23.5M语义关系及9.7M指称语句,并基准测试,助力鲁棒导航。

摘要:

随着大型语言模型(LLMs)、视觉语言模型(VLMs)及其他通用基础模型的近期兴起,多模态、多任务的具身智能体在仅以自然语言作为输入的情况下,能够在多样化环境中运行的潜力日益增长。其中一个应用领域便是使用自然语言指令进行室内导航。然而,尽管近期取得了进展,该问题仍因所需的空间推理和语义理解而具有挑战性,尤其是在可能包含许多属于细粒度类别的物体的任意场景中。为应对这一挑战,我们构建了用于3D场景中视觉与语言引导动作的最大真实世界数据集(VLA-3D),该数据集包含来自现有数据集的超过11.5K个扫描的3D室内房间、23.5M条启发式生成的物体间语义关系,以及9.7M条合成生成的指称语句。我们的数据集由处理后的3D点云、语义物体和房间标注、场景图、可导航自由空间标注以及指称语言语句组成,这些语句特别关注用于消歧物体的视角无关空间关系。这些特征的目标是辅助下游导航任务,尤其是在真实世界系统中,在变化场景和不完美语言的开放世界中必须保证一定程度的鲁棒性。我们使用当前最先进的模型对我们的数据集进行基准测试,以获得性能基线。所有用于生成和可视化数据集的代码均已公开,见https://github.com/HaochenZ11/VLA-3D。通过发布该数据集,我们希望为语义3D场景理解的进展提供资源,使其对变化具有鲁棒性,并有助于交互式室内导航系统的开发。

英文摘要:

With the recent rise of Large Language Models (LLMs), Vision-Language Models (VLMs), and other general foundation models, there is growing potential for multimodal, multi-task embodied agents that can operate in diverse environments given only natural language as input. One such application area is indoor navigation using natural language instructions. However, despite recent progress, this problem remains challenging due to the spatial reasoning and semantic understanding required, particularly in arbitrary scenes that may contain many objects belonging to fine-grained classes. To address this challenge, we curate the largest real-world dataset for Vision and Language-guided Action in 3D Scenes (VLA-3D), consisting of over 11.5K scanned 3D indoor rooms from existing datasets, 23.5M heuristically generated semantic relations between objects, and 9.7M synthetically generated referential statements. Our dataset consists of processed 3D point clouds, semantic object and room annotations, scene graphs, navigable free space annotations, and referential language statements that specifically focus on view-independent spatial relations for disambiguating objects. The goal of these features is to aid the downstream task of navigation, especially on real-world systems where some level of robustness must be guaranteed in an open world of changing scenes and imperfect language. We benchmark our dataset with current state-of-the-art models to obtain a performance baseline. All code to generate and visualize the dataset is publicly released, see https://github.com/HaochenZ11/VLA-3D. With the release of this dataset, we hope to provide a resource for progress in semantic 3D scene understanding that is robust to changes and one which will aid the development of interactive indoor navigation systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑