arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.17320cs.RO
收藏

在基于强化学习的足式运动控制中利用对称性

Leveraging Symmetry in RL-based Legged Locomotion Control

UC Berkeley(加州大学伯克利分校);Istituto Italiano di Tecnologia(意大利技术研究院);Shanghai Qi Zhi Institute(上海期智研究院)

Zhi Su, Xiaoyu Huang, Daniel Ordoñez-Apraez, Yunfei Li, Zhongyu Li, Qiayuan Liao, Giulio Turrisi, Massimiliano Pontil, Claudio Semini, Yi Wu, Koushil Sreenath

AI总结:

本文研究如何利用形态对称性改进足式机器人的无模型强化学习,比较严格等变网络与数据增强方法,发现严格等变策略在样本效率、步态质量和真实硬件零样本部署上表现最佳。

摘要:

无模型强化学习是自主解决具有挑战性的机器人控制问题的一种有前景的方法,但在缺乏机器人运动学和动力学形态信息的情况下面临探索困难。对具有对称状态的多种模态探索不足,会导致行为往往不自然且次优。这一问题在具有形态对称性的机器人系统中尤为突出,例如足式机器人,由此产生的不对称和非周期性行为会损害性能、鲁棒性以及向真实硬件的可迁移性。为了缓解这一挑战,我们可以利用对称性,通过等变性/不变性约束来引导并改进策略学习中的探索。在本文中,我们研究了两种引入对称性的方法的有效性:修改网络架构使其严格等变/不变,以及利用数据增强来近似等变/不变的演员-评论家。我们在具有挑战性的运动-操作和双足运动任务上实现了这些方法,并与无约束基线进行了比较。我们发现,严格等变的策略在仿真中的样本效率和任务性能方面始终优于其他方法。此外,引入对称性的方法表现出更好的步态质量、更高的鲁棒性,并且可以在真实世界实验中零样本部署。

英文摘要:

Model-free reinforcement learning is a promising approach for autonomously solving challenging robotics control problems, but faces exploration difficulty without information of the robot's kinematics and dynamics morphology. The under-exploration of multiple modalities with symmetric states leads to behaviors that are often unnatural and sub-optimal. This issue becomes particularly pronounced in the context of robotic systems with morphological symmetries, such as legged robots for which the resulting asymmetric and aperiodic behaviors compromise performance, robustness, and transferability to real hardware. To mitigate this challenge, we can leverage symmetry to guide and improve the exploration in policy learning via equivariance/invariance constraints. In this paper, we investigate the efficacy of two approaches to incorporate symmetry: modifying the network architectures to be strictly equivariant/invariant, and leveraging data augmentation to approximate equivariant/invariant actor-critics. We implement the methods on challenging loco-manipulation and bipedal locomotion tasks and compare with an unconstrained baseline. We find that the strictly equivariant policy consistently outperforms other methods in sample efficiency and task performance in simulation. In addition, symmetry-incorporated approaches exhibit better gait quality, higher robustness and can be deployed zero-shot in real-world experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑