基于关系场景图的对象接地自然语言指令
Relational Scene Graphs for Object Grounding of Natural Language Commands
AI总结：
本文提出基于关系场景图的方法，通过结合LLM和VLM增强自然语言指令中对象接地的准确性。
摘要：
机器人正在越来越多地应用于人类环境，这增加了对自然人机交互的需求。然而，理解自然语言指令需要机器人推断出意图的任务以及如何将其分解为可执行的动作，并将这些动作接地在机器人对环境的知识中，包括相关对象、代理和位置。这一挑战可以通过结合大型语言模型（LLMs）理解自然语言的能力与3D场景图（3DSGs）将推断出的动作接地在环境的语义表示中来解决。然而，许多3DSGs缺乏对象之间的显式空间关系，尽管人类经常依靠这些关系来描述环境。本文研究了将开放或封闭词汇空间关系纳入3DSGs是否能提高LLMs解读自然语言指令的能力。为此，我们提出了一条基于LLM的从开放词汇语言指令中进行目标对象接地的管道，以及一条基于视觉语言模型（VLM）的管道，用于从映射过程中捕获的图像中添加开放词汇空间边到3DSGs中。最后，两个LLM在评估其在目标对象接地下游任务上的性能的研究中被评估。我们的研究证明，显式空间关系提高了LLMs对对象的接地能力。此外，使用VLM进行开放词汇关系生成从机器人捕获的图像中证明是可行的，但其相对于封闭词汇关系的优势被发现是有限的。
英文摘要：
Robots are finding wider adoption in human environments, increasing the need for natural human-robot interaction. However, understanding a natural language command requires the robot to infer the intended task and how to decompose it into executable actions, and to ground those actions in the robot's knowledge of the environment, including relevant objects, agents, and locations. This challenge can be addressed by combining the capabilities of large language models (LLMs) to understand natural language with 3D scene graphs (3DSGs) for grounding inferred actions in a semantic representation of the environment. However, many 3DSGs lack explicit spatial relations between objects, even though humans often rely on these relations to describe an environment. This paper investigates whether incorporating open- or closed-vocabulary spatial relations into 3DSGs can improve the ability of LLMs to interpret natural language commands. To address this, we implement two pipelines using off-the-shelf models: an LLM-based pipeline for target object grounding from open-vocabulary language commands and a vision language model-based pipeline to add open-vocabulary spatial edges to 3DSGs from images captured while mapping. Finally, we evaluate two LLMs across 14 scenes using 905 natural language statements (786 procedurally-generated, 119 human-authored) to assess performance on the downstream task of target object grounding. Our study demonstrates that explicit spatial relations improve the ability of LLMs to ground objects, and while open-vocabulary relation generation with vision language models proves feasible from robot-captured images, our analysis did not yield evidence favoring either open- or closed-vocabulary relations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
