面向检索增强推理的开放世界3D场景图生成
Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning
AI总结：
针对开放世界3D场景理解的闭词汇监督和静态标注局限，提出集成视觉语言模型（VLMs）与检索推理的统一框架，通过动态场景图生成和检索增强推理，在多任务中实现稳健泛化与优越性能。
摘要：
在开放世界环境中理解3D场景对视觉和机器人学构成了根本性挑战，尤其是由于闭词汇监督和静态标注的局限性。为此，我们提出了一个面向检索增强推理的开放世界3D场景图生成统一框架，该框架支持可泛化且交互式的3D场景理解。我们的方法将视觉语言模型（VLMs）与基于检索的推理相结合，以支持多模态探索和语言引导的交互。该框架包含两个关键组件：（1）动态场景图生成模块，无需固定标签集即可检测对象并推断语义关系；（2）检索增强推理管道，将场景图编码到向量数据库中以支持文本/图像条件查询。我们在3DSSG和Replica基准上对场景问答、视觉定位、实例检索和任务规划这四个任务进行了评估，证明了在不同环境中的稳健泛化能力和优越性能。我们的结果强调了将开放词汇感知与基于检索的推理相结合对可扩展3D场景理解的有效性。
英文摘要：
Understanding 3D scenes in open-world settings poses fundamental challenges for vision and robotics, particularly due to the limitations of closed-vocabulary supervision and static annotations. To address this, we propose a unified framework for Open-World 3D Scene Graph Generation with Retrieval-Augmented Reasoning, which enables generalizable and interactive 3D scene understanding. Our method integrates Vision-Language Models (VLMs) with retrieval-based reasoning to support multimodal exploration and language-guided interaction. The framework comprises two key components: (1) a dynamic scene graph generation module that detects objects and infers semantic relationships without fixed label sets, and (2) a retrieval-augmented reasoning pipeline that encodes scene graphs into a vector database to support text/image-conditioned queries. We evaluate our method on 3DSSG and Replica benchmarks across four tasks-scene question answering, visual grounding, instance retrieval, and task planning-demonstrating robust generalization and superior performance in diverse environments. Our results highlight the effectiveness of combining open-vocabulary perception with retrieval-based reasoning for scalable 3D scene understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
