First-order Policy Gradient (FoPG) algorithms such as Backpropagation through Time and Analytical Policy Gradients leverage local simulation physics to accelerate policy search, significantly improving sample efficiency in robot control compared to standard model-free reinforcement learning. However, FoPG algorithms can exhibit poor learning dynamics in contact-rich tasks like locomotion. Previous approaches address this issue by alleviating contact dynamics via algorithmic or simulation innovations. In contrast, we propose guiding the policy search by learning a residual over a simple baseline policy. For quadruped locomotion, we find that the role of residual policy learning in FoPG-based training (FoPG RPL) is primarily to improve asymptotic rewards, compared to improving sample efficiency for model-free RL. Additionally, we provide insights on applying FoPG's to pixel-based local navigation, training a point-mass robot to convergence within seconds. Finally, we showcase the versatility of FoPG RPL by using it to train locomotion and perceptive navigation end-to-end on a quadruped in minutes.
Learning robust perceptive locomotion for quadrupedal robots in the wild
Robotic Systems Lab, ETH Zurich(苏黎世联邦理工学院机器人系统实验室);Robotics and Artificial Intelligence Lab, KAIST(韩国科学技术院机器人与人工智能实验室);Intelligent Systems Lab, Intel(英特尔智能系统实验室)
Motion Priors Reimagined: Adapting Flat-Terrain Skills for Complex Quadruped Mobility
Department of Mechanical Engineering, EPFL(瑞士联邦理工学院机械工程系);ETH AI Center, ETH Zurich(苏黎世联邦理工学院人工智能中心);Robotic Systems Lab, ETH Zurich(苏黎世联邦理工学院机器人系统实验室)
DiffSim2Real: Deploying Quadrupedal Locomotion Policies Purely Trained in Differentiable Simulation
Robotic Systems Lab, ETH Zürich(苏黎世联邦理工学院机器人系统实验室);Georgia Institute of Technology(佐治亚理工学院);The AI Institute(人工智能研究所);Institute for Research in Technology, ICAI School of Engineering, Comillas Pontifical University(科米利亚斯主教大学技术研究所(ICAI工程学院))
First Steps: Latent-Space Control with Semantic Constraints for Quadruped Locomotion
Applied AI Lab (A2I)(应用人工智能实验室(A2I));Dynamic Robot Systems (DRS)(动态机器人系统(DRS));Oxford Robotics Institute (ORI), University of Oxford(牛津大学牛津机器人研究所(ORI))
Learning Quadruped Locomotion using Bio-Inspired Neural Networks with Intrinsic Rhythmicity
Shenzhen Amigaga Technology Co Ltd.(深圳阿米加加科技有限公司);Sun Yat-sen University(中山大学);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);University College London(伦敦大学学院)
DiffCoTune: Differentiable Co-Tuning for Cross-domain Robot Control
Department of Aerospace and Mechanical Engineering, USC Viterbi School of Engineering(航空航天与机械工程系,USC维特比工程学院);Department of Mechanical Science and Engineering, University of Illinois Urbana-Champaign(机械科学与工程系,伊利诺伊大学厄巴纳-香槟分校)