集成 LMM 规划器与 3D 技能策略以实现可泛化操作
Integrating LMM Planners and 3D Skill Policies for Generalizable Manipulation
AI总结：
提出 LMM-3DP 框架，通过联合注意力机制在 3D transformer 中无缝集成 LMM 高层规划与语义感知 3D 特征场低层控制，在真实厨房环境中显著提升了长周期任务的成功率与规划准确率。
摘要：
大型多模态模型（LMMs）在视觉推理能力上的最新进展，以及 3D 特征场的语义丰富化，拓展了机器人能力的边界。这些发展为弥合 LMMs 的高层推理与利用 3D 特征场的低层控制策略之间的差距带来了巨大潜力。在本研究中，我们提出了 LMM-3DP，这是一个能够集成 LMM 规划器与 3D 技能策略（3D skill Policies）的框架。我们的方法包含三个关键视角：高层规划、低层控制与有效集成。在高层规划方面，LMM-3DP 支持针对环境干扰的动态场景理解、带有自我反馈的评估智能体（critic agent）、历史策略记忆以及失败后的重试机制。在低层控制方面，LMM-3DP 利用语义感知的 3D 特征场来实现精准操作。在为实现机器人动作而对高层与低层控制进行对齐时，代表高层策略的语言嵌入在 3D transformer 中与 3D 特征场进行联合注意力处理，以实现无缝集成。我们在真实世界的厨房环境中，对多种技能和长周期任务广泛评估了该方法。结果显示，与基于 LLM 的基线相比，低层控制的成功率显著提升了 1.45 倍，高层规划的准确率提升了约 1.5 倍。演示视频和 LMM-3DP 的概览可在 https://lmm-3dp-release.github.io 获取。
英文摘要：
The recent advancements in visual reasoning capabilities of large multimodal models (LMMs) and the semantic enrichment of 3D feature fields have expanded the horizons of robotic capabilities. These developments hold significant potential for bridging the gap between high-level reasoning from LMMs and low-level control policies utilizing 3D feature fields. In this work, we introduce LMM-3DP, a framework that can integrate LMM planners and 3D skill Policies. Our approach consists of three key perspectives: high-level planning, low-level control, and effective integration. For high-level planning, LMM-3DP supports dynamic scene understanding for environment disturbances, a critic agent with self-feedback, history policy memorization, and reattempts after failures. For low-level control, LMM-3DP utilizes a semantic-aware 3D feature field for accurate manipulation. In aligning high-level and low-level control for robot actions, language embeddings representing the high-level policy are jointly attended with the 3D feature field in the 3D transformer for seamless integration. We extensively evaluate our approach across multiple skills and long-horizon tasks in a real-world kitchen environment. Our results show a significant 1.45x success rate increase in low-level control and an approximate 1.5x improvement in high-level planning accuracy compared to LLM-based baselines. Demo videos and an overview of LMM-3DP are available at https://lmm-3dp-release.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
