arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.03076cs.RO
收藏

使用可微仿真进行感知四足控制的残差策略学习

Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation

ETH Zurich(苏黎世联邦理工学院);University of Zurich(苏黎世大学);National University of Singapore(新加坡国立大学)

Jing Yuan Luo, Yunlong Song, Victor Klemm, Fan Shi, Davide Scaramuzza, Marco Hutter

AI总结:

本文提出在简单基线策略上学习残差以引导一阶策略梯度训练,改善接触丰富任务中的学习动态,并实现四足机器人运动与感知导航的快速端到端训练。

摘要:

一阶策略梯度(FoPG)算法,如时间反向传播和解析策略梯度,利用局部仿真物理来加速策略搜索,与标准的无模型强化学习相比,显著提高了机器人控制的样本效率。然而,FoPG 算法在诸如运动这样的接触丰富任务中可能表现出较差的学习动态。先前的方法通过算法或仿真创新来缓解接触动态问题。与之不同,我们提出通过在一个简单的基线策略上学习残差来引导策略搜索。对于四足运动,我们发现残差策略学习在基于 FoPG 的训练(FoPG RPL)中的作用主要是提高渐近奖励,而不是像无模型强化学习那样提高样本效率。此外,我们提供了将 FoPG 应用于基于像素的局部导航的见解,在几秒钟内训练一个点质量机器人达到收敛。最后,我们展示了 FoPG RPL 的多功能性,利用它在几分钟内端到端地训练四足机器人的运动和感知导航。

英文摘要:

First-order Policy Gradient (FoPG) algorithms such as Backpropagation through Time and Analytical Policy Gradients leverage local simulation physics to accelerate policy search, significantly improving sample efficiency in robot control compared to standard model-free reinforcement learning. However, FoPG algorithms can exhibit poor learning dynamics in contact-rich tasks like locomotion. Previous approaches address this issue by alleviating contact dynamics via algorithmic or simulation innovations. In contrast, we propose guiding the policy search by learning a residual over a simple baseline policy. For quadruped locomotion, we find that the role of residual policy learning in FoPG-based training (FoPG RPL) is primarily to improve asymptotic rewards, compared to improving sample efficiency for model-free RL. Additionally, we provide insights on applying FoPG's to pixel-based local navigation, training a point-mass robot to convergence within seconds. Finally, we showcase the versatility of FoPG RPL by using it to train locomotion and perceptive navigation end-to-end on a quadruped in minutes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑