可查询3D场景表示：一种面向语义推理与机器人任务规划的多模态框架
Queryable 3D Scene Representation: A Multi-Modal Framework for Semantic Reasoning and Robotic Task Planning
AI总结：
针对机器人场景理解需求，提出融合三种3D表示、结合大视觉语言模型的3D QSR框架，支持语义查询与任务规划，经模拟及真实湿实验室数字孪生场景验证有效。
摘要：
为使机器人能够理解人类高层指令并执行复杂任务，核心挑战在于实现全面的场景理解：即以有意义的方式解读3D环境并与之交互。这需要一种融合精确几何结构与丰富的人类可理解语义的智能地图。为解决这一问题，我们提出了3D可查询场景表示（3D Queryable Scene Representation，简称3D QSR），这是一种基于多模态数据构建的新型框架，统一了三种互补的3D表示形式：（1）来自全景重建的具备3D一致性的新视角渲染与分割结果；（2）来自3D点云的精确几何信息；（3）通过3D场景图实现的结构化、可扩展组织方式。该框架以对象为中心设计，与大视觉语言模型集成，通过关联多模态对象嵌入实现语义可查询性，并支持对象级别的几何、视觉与语义信息检索。检索到的数据随后被加载到机器人任务规划器中用于下游执行。我们在Unity中通过模拟机器人任务规划场景对方法进行评估，场景以抽象语言指令为引导，使用室内公开数据集Replica。此外，我们将其应用于真实湿实验室环境的数字孪生场景中，测试QSR支持的应急响应机器人任务规划。结果表明，该框架能够促进场景理解，整合空间与语义推理，可在复杂3D环境中有效将人类高层指令转化为精确的机器人任务规划。
英文摘要：
To enable robots to comprehend high-level human instructions and perform complex tasks, a key challenge lies in achieving comprehensive scene understanding: interpreting and interacting with the 3D environment in a meaningful way. This requires a smart map that fuses accurate geometric structure with rich, human-understandable semantics. To address this, we introduce the 3D Queryable Scene Representation (3D QSR), a novel framework built on multimedia data that unifies three complementary 3D representations: (1) 3D-consistent novel view rendering and segmentation from panoptic reconstruction, (2) precise geometry from 3D point clouds, and (3) structured, scalable organization via 3D scene graphs. Built on an object-centric design, the framework integrates with large vision-language models to enable semantic queryability by linking multimodal object embeddings, and supporting object-level retrieval of geometric, visual, and semantic information. The retrieved data are then loaded into a robotic task planner for downstream execution. We evaluate our approach through simulated robotic task planning scenarios in Unity, guided by abstract language instructions and using the indoor public dataset Replica. Furthermore, we apply it in a digital duplicate of a real wet lab environment to test QSR-supported robotic task planning for emergency response. The results demonstrate the framework's ability to facilitate scene understanding and integrate spatial and semantic reasoning, effectively translating high-level human instructions into precise robotic task planning in complex 3D environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
