arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.12263cs.RO
收藏

面向全身操作的规划引导强化学习

Plan-Guided Reinforcement Learning for Whole-Body Manipulation

University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Toyota Research Institute(丰田研究院)

Mengchao Zhang, Jose Barreiros, Aykut Ozgun Onol

AI总结:

提出规划引导强化学习(PGRL)框架,结合基于模型的规划与RL优势,在Punyo机器人上实现大箱子枢转与举起的全身操作,减少人工监督并提升策略鲁棒性。

摘要:

由于接触交互规划中固有的组合学快速增长,合成复杂的全身操作行为存在根本性挑战。基于模型的方法在解决长时程操作任务方面显示出可喜的结果,但它们通常在严格的假设下工作,例如已知的模型参数、对环境状态的预言式观察以及简化的动力学,导致生成的规划难以轻松迁移到硬件上。基于学习的方法,如模仿学习(IL)和强化学习(RL),已被证明在处理分布内状态时具有鲁棒性;然而,它们需要大量的人工监督。具体而言,无模型RL需要繁琐的奖励塑形过程。另一方面,IL方法依赖于涉及高级遥操作方法的人类演示。在这项工作中,我们提出了一个规划引导强化学习(PGRL)框架,以结合基于模型的规划和强化学习的优势。我们的方法需要最少的人工监督,因为它依赖于基于模型的规划器生成的规划来引导RL中的探索。作为交换,得益于域随机化,RL推导出了更鲁棒的策略。我们在Punyo(一种带有柔性充气手臂覆盖物的上半身人形机器人)上测试了这种方法,执行全身操作任务,以枢转并举起一个大箱子。我们的初步结果表明,所提出的方法有望解决对单独的基于模型或基于学习的策略而言仍然困难的挑战。

英文摘要:

Synthesizing complex whole-body manipulation behaviors has fundamental challenges due to the rapidly growing combinatorics inherent to contact interaction planning. While model-based methods have shown promising results in solving long-horizon manipulation tasks, they often work under strict assumptions, such as known model parameters, oracular observation of the environment state, and simplified dynamics, resulting in plans that cannot easily transfer to hardware. Learning-based approaches, such as imitation learning (IL) and reinforcement learning (RL), have been shown to be robust when operating over in-distribution states; however, they need heavy human supervision. Specifically, model-free RL requires a tedious reward-shaping process. IL methods, on the other hand, rely on human demonstrations that involve advanced teleoperation methods. In this work, we propose a plan-guided reinforcement learning (PGRL) framework to combine the advantages of model-based planning and reinforcement learning. Our method requires minimal human supervision because it relies on plans generated by model-based planners to guide the exploration in RL. In exchange, RL derives a more robust policy thanks to domain randomization. We test this approach on a whole-body manipulation task on Punyo, an upper-body humanoid robot with compliant, air-filled arm coverings, to pivot and lift a large box. Our preliminary results indicate that the proposed methodology is promising to address challenges that remain difficult for either model- or learning-based strategies alone.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑