arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.20154cs.RO
收藏

GravMAD：用于泛化三维操作的、基于接地空间价值地图引导的动作扩散

GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation

State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Intelligence Science and Technology, Nanjing University (Suzhou Campus)（南京大学智能科学与技术学院（苏州校区））；School of Computer Engineering and Science, Shanghai University（上海大学计算机工程与科学学院）

Yangtao Chen, Zixuan Chen, Junhui Yin, Jing Huo, Pinzhuo Tian, Jieqi Shi, Yang Gao

备注：
ICLR 2025. The first two authors contributed equally
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出语言条件、子目标驱动的动作扩散框架 GravMAD，利用演示发现与基础模型生成 GravMaps 空间引导，在 RLBench 和真实机器人上显著提升三维操作泛化性能。

摘要：

机器人遵循语言指令并执行多样化三维操作任务的能力在机器人学习中至关重要。传统基于模仿学习的方法在已见任务上表现良好，但由于任务的可变性，在处理新颖、未见任务时面临困难。近期方法利用大型基础模型辅助理解新颖任务，从而缓解这一问题。然而，这些方法缺乏针对具体任务的学习过程，而该过程对于准确理解三维环境必不可少，因此往往导致执行失败。本文提出 GravMAD，这是一个由子目标驱动、以语言为条件的动作扩散框架，结合了模仿学习与基础模型的优势。该方法根据语言指令将任务分解为子目标，从而在训练和推理期间都能提供辅助引导。训练阶段，我们引入 Sub-goal Keypose Discovery（子目标关键姿态发现），从演示中识别关键子目标。推理与训练不同，因为此时没有可用演示，所以我们使用预训练基础模型弥合差距，并为当前任务识别子目标。在两个阶段中，都会由子目标生成 GravMaps，与固定三维位置相比，为 GravMAD 提供更灵活的三维空间引导。在 RLBench 上的实证评估表明，GravMAD 显著优于最先进方法：在新颖任务上提升 28.63%，在训练期间遇到的任务上提升 13.36%。在真实机器人任务上的评估进一步表明，GravMAD 能够对真实世界任务进行推理，将其与相关视觉信息关联，并泛化到新颖任务。这些结果证明了 GravMAD 在三维操作中强大的多任务学习与泛化能力。视频演示见：https://gravmad.github.io。

英文摘要：

Robots' ability to follow language instructions and execute diverse 3D manipulation tasks is vital in robot learning. Traditional imitation learning-based methods perform well on seen tasks but struggle with novel, unseen ones due to variability. Recent approaches leverage large foundation models to assist in understanding novel tasks, thereby mitigating this issue. However, these methods lack a task-specific learning process, which is essential for an accurate understanding of 3D environments, often leading to execution failures. In this paper, we introduce GravMAD, a sub-goal-driven, language-conditioned action diffusion framework that combines the strengths of imitation learning and foundation models. Our approach breaks tasks into sub-goals based on language instructions, allowing auxiliary guidance during both training and inference. During training, we introduce Sub-goal Keypose Discovery to identify key sub-goals from demonstrations. Inference differs from training, as there are no demonstrations available, so we use pre-trained foundation models to bridge the gap and identify sub-goals for the current task. In both phases, GravMaps are generated from sub-goals, providing GravMAD with more flexible 3D spatial guidance compared to fixed 3D positions. Empirical evaluations on RLBench show that GravMAD significantly outperforms state-of-the-art methods, with a 28.63% improvement on novel tasks and a 13.36% gain on tasks encountered during training. Evaluations on real-world robotic tasks further show that GravMAD can reason about real-world tasks, associate them with relevant visual information, and generalize to novel tasks. These results demonstrate GravMAD's strong multi-task learning and generalization in 3D manipulation. Video demonstrations are available at: https://gravmad.github.io.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑