arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.16643cs.CVcs.AIcs.RO
收藏

用于3D场景图自动推理的结构化接口

Structured Interfaces for Automated Reasoning with 3D Scene Graphs

Massachusetts Institute of Technology(麻省理工学院)

Aaron Ray, Jacob Arkin, Harel Biggie, Chuchu Fan, Luca Carlone, Nicholas Roy

AI总结:

针对LLM处理大型3D场景图时上下文序列化难以扩展的问题,本文提出基于图数据库和Cypher查询接口的检索增强方法,显著提升指令遵循与场景问答性能并减少token消耗。

摘要:

为了使机器人具备理解用户自然语言输入并对其作出反应的能力,必须将自然语言与机器人对世界的底层表征连接起来。近来,大语言模型(LLM)和3D场景图(3DSG)已成为自然语言 grounding 和世界表征的流行选择。在本研究中,我们应对将LLM与3DSG结合用于自然语言 grounding 的挑战。现有方法将场景图编码为LLM上下文窗口内的序列化文本,但这种编码无法扩展到大型或信息丰富的3DSG。相反,我们提出使用一种检索增强生成形式,以选择与任务相关的3DSG子集。我们将3DSG编码到图数据库中,并向LLM提供查询语言接口(Cypher)作为工具,使其能够检索用于语言 grounding 的相关数据。我们在指令遵循和场景问答任务上评估了该方法,并与基线上下文窗口方法和代码生成方法进行比较。结果表明,在本地模型和基于云端的模型上,使用Cypher作为3D场景图接口都能显著更好地扩展到大型、信息丰富的图。这不仅在 grounded 语言任务上带来大幅性能提升,还显著减少了场景图内容的token数量。视频补充材料可在 https://www.youtube.com/watch?v=zY_YI9giZSA 获取。

英文摘要:

In order to provide a robot with the ability to understand and react to a user's natural language inputs, the natural language must be connected to the robot's underlying representations of the world. Recently, large language models (LLMs) and 3D scene graphs (3DSGs) have become a popular choice for grounding natural language and representing the world. In this work, we address the challenge of using LLMs with 3DSGs to ground natural language. Existing methods encode the scene graph as serialized text within the LLM's context window, but this encoding does not scale to large or rich 3DSGs. Instead, we propose to use a form of Retrieval Augmented Generation to select a subset of the 3DSG relevant to the task. We encode a 3DSG in a graph database and provide a query language interface (Cypher) as a tool to the LLM with which it can retrieve relevant data for language grounding. We evaluate our approach on instruction following and scene question-answering tasks and compare against baseline context window and code generation methods. Our results show that using Cypher as an interface to 3D scene graphs scales significantly better to large, rich graphs on both local and cloud-based models. This leads to large performance improvements in grounded language tasks while also substantially reducing the token count of the scene graph content. A video supplement is available at https://www.youtube.com/watch?v=zY_YI9giZSA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑