arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.15714cs.CV

ROOT：基于VLM的室内场景理解及其拓展系统

ROOT: VLM based System for Indoor Scene Understanding and Beyond

University of Science and Technology of China（中国科学技术大学）；Game AI Center, Tencent IEG（腾讯互动娱乐事业群游戏AI中心）

Yonghui Wang, Shi-Yong Chen, Zhenxing Zhou, Siyi Li, Haoran Li, Wengang Zhou, Houqiang Li

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于VLM的ROOT系统，通过迭代物体感知、SceneVLM空间层次场景图与距离估计，结合超61万张室内图像训练，提升室内场景理解并支持3D生成与具身智能等下游任务。

摘要：

近年来，视觉语言模型（VLMs）取得了显著进展，但这些模型在室内场景的空间层次推理方面仍面临挑战。在本研究中，我们提出了ROOT，一个基于VLM的系统，旨在增强对室内场景的分析。具体而言，我们首先开发了一种使用GPT-4V的迭代式物体感知算法，用于检测室内场景中的物体实体。随后，利用视觉基础模型获取关于场景的额外元信息，例如边界框。在此基础数据之上，我们提出了一种专用的VLM——SceneVLM，它能够生成空间层次场景图，并提供室内环境中物体之间的距离信息。这些信息增强了我们对室内场景空间布局的理解。为了训练我们的SceneVLM，我们从多个公开室内数据集中收集了超过610,000张图像，并实现了一个带有半自动化技术的场景数据生成流程，以建立室内物体之间的关系并估计它们之间的距离。通过利用这些丰富的数据，我们实施了多种训练方案并完成了SceneVLM。我们的实验表明，ROOT有助于室内场景理解，并在多种下游应用中证明有效，例如3D场景生成和具身人工智能。代码将在https://github.com/harrytea/ROOT发布。

英文摘要：

Recently, Vision Language Models (VLMs) have experienced significant advancements, yet these models still face challenges in spatial hierarchical reasoning within indoor scenes. In this study, we introduce ROOT, a VLM-based system designed to enhance the analysis of indoor scenes. Specifically, we first develop an iterative object perception algorithm using GPT-4V to detect object entities within indoor scenes. This is followed by employing vision foundation models to acquire additional meta-information about the scene, such as bounding boxes. Building on this foundational data, we propose a specialized VLM, SceneVLM, which is capable of generating spatial hierarchical scene graphs and providing distance information for objects within indoor environments. This information enhances our understanding of the spatial arrangement of indoor scenes. To train our SceneVLM, we collect over 610,000 images from various public indoor datasets and implement a scene data generation pipeline with a semi-automated technique to establish relationships and estimate distances among indoor objects. By utilizing this enriched data, we conduct various training recipes and finish SceneVLM. Our experiments demonstrate that \rootname facilitates indoor scene understanding and proves effective in diverse downstream applications, such as 3D scene generation and embodied AI. The code will be released at \url{https://github.com/harrytea/ROOT}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑