开放词汇的功能性3D场景图用于真实世界室内空间
Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces
AI总结：
本文提出从RGB-D图像预测功能性3D场景图，利用基础模型编码功能知识，在扩展数据集上显著优于基线，并支持3D问答与机器人操作。
摘要：
我们引入了从带姿态的RGB-D图像预测真实世界室内环境的功能性3D场景图的任务。与侧重于物体空间关系的传统3D场景图不同，功能性3D场景图捕捉物体、交互元素及其功能关系。由于缺乏训练数据，我们利用基础模型，包括视觉语言模型（VLMs）和大语言模型（LLMs），来编码功能知识。我们在扩展的SceneFun3D数据集和新收集的数据集FunGraph3D上评估了我们的方法，两者均标注了功能性3D场景图。我们的方法显著优于改编的基线方法，包括Open3DSG和ConceptGraph，展示了其在建模复杂场景功能方面的有效性。我们还展示了使用功能性3D场景图的下游应用，如3D问答和机器人操作。请参阅我们的项目页面：https://openfungraph.github.io
英文摘要：
We introduce the task of predicting functional 3D scene graphs for real-world indoor environments from posed RGB-D images. Unlike traditional 3D scene graphs that focus on spatial relationships of objects, functional 3D scene graphs capture objects, interactive elements, and their functional relationships. Due to the lack of training data, we leverage foundation models, including visual language models (VLMs) and large language models (LLMs), to encode functional knowledge. We evaluate our approach on an extended SceneFun3D dataset and a newly collected dataset, FunGraph3D, both annotated with functional 3D scene graphs. Our method significantly outperforms adapted baselines, including Open3DSG and ConceptGraph, demonstrating its effectiveness in modeling complex scene functionalities. We also demonstrate downstream applications such as 3D question answering and robotic manipulation using functional 3D scene graphs. See our project page at https://openfungraph.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
