开放词汇移动操作在未知动态环境中基于3D语义地图的应用
Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Maps
AI总结：
针对未知动态环境中的开放词汇移动操作，提出结合预训练视觉语言模型和3D语义地图的框架，利用大语言模型进行规划，在JSR-1机器人上实现高成功率并支持失败重规划。
摘要：
开放词汇移动操作（OVMM）是自主机器人的关键能力，尤其是在面对未知和动态环境带来的挑战时。该任务要求机器人探索并建立对周围环境的语义理解，生成可行的计划以实现操作目标，适应环境变化，并理解人类的自然语言指令。为解决这些挑战，我们提出了一种新颖的框架，该框架利用预训练视觉语言模型（VLMs）的零样本检测和接地识别能力，结合密集3D实体重建来构建3D语义地图。此外，我们利用大型语言模型（LLMs）进行空间区域抽象和在线规划，融合人类指令和空间语义上下文。我们构建了一个10自由度移动操作机器人平台JSR-1，并在真实世界机器人实验中证明，所提出的框架能够有效捕获空间语义并处理自然语言用户指令，以在动态环境设置下完成零样本OVMM任务，在105个回合中总体导航和任务成功率分别为80.95%和73.33%，与基线相比，SFT和SPL分别提高了157.18%和19.53%。此外，当初始计划失败时，该框架能够基于从3D语义地图获得的空间语义上下文，重新规划至下一个最可能的候选位置，保持平均成功率为76.67%。
英文摘要：
Open-Vocabulary Mobile Manipulation (OVMM) is a crucial capability for autonomous robots, especially when faced with the challenges posed by unknown and dynamic environments. This task requires robots to explore and build a semantic understanding of their surroundings, generate feasible plans to achieve manipulation goals, adapt to environmental changes, and comprehend natural language instructions from humans. To address these challenges, we propose a novel framework that leverages the zero-shot detection and grounded recognition capabilities of pretraining visual-language models (VLMs) combined with dense 3D entity reconstruction to build 3D semantic maps. Additionally, we utilize large language models (LLMs) for spatial region abstraction and online planning, incorporating human instructions and spatial semantic context. We have built a 10-DoF mobile manipulation robotic platform JSR-1 and demonstrated in real-world robot experiments that our proposed framework can effectively capture spatial semantics and process natural language user instructions for zero-shot OVMM tasks under dynamic environment settings, with an overall navigation and task success rate of 80.95% and 73.33% over 105 episodes, and better SFT and SPL by 157.18% and 19.53% respectively compared to the baseline. Furthermore, the framework is capable of replanning towards the next most probable candidate location based on the spatial semantic context derived from the 3D semantic map when initial plans fail, keeping an average success rate of 76.67%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
