arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.01962cs.RO
收藏

一种无需强化学习的高效基于模型敏捷运动技能学习方法

An Efficient Model-Based Approach on Learning Agile Motor Skills without Reinforcement

Tencent Robotics X(腾讯Robotics X实验室);Chinese University of Hong Kong(香港中文大学);Southern University of Science and Technology(南方科技大学);The Chinese University of Hong Kong(香港中文大学);University of Alberta(阿尔伯塔大学)

Haojie Shi, Tingguang Li, Qingxu Zhu, Jiapeng Sheng, Lei Han, Max Q. -H. Meng

AI总结:

提出一种结合世界模型与VAE策略网络的高效基于模型框架,无需强化学习即可学习敏捷运动技能,实现十倍样本效率提升和快速真实世界部署。

摘要:

基于学习的方法通过深度强化学习提高了四足机器人的运动技能。然而,仿真到现实的差距和低样本效率仍然限制了技能迁移。为解决这一问题,我们提出了一种高效基于模型的学习框架,该框架将世界模型与策略网络相结合。我们训练一个可微分的世界模型来预测未来状态,并利用它直接监督基于变分自编码器(VAE)的策略网络以模仿真实动物行为。这显著减少了对真实交互数据的需求,并允许快速更新策略。我们还开发了一个高层网络来跟踪多样化的指令和轨迹。我们的仿真结果显示,与PPO等强化学习方法相比,样本效率提高了十倍。在真实世界测试中,我们的策略仅需两分钟的数据收集时间即可实现熟练的指令跟随性能,并能很好地泛化到新的速度和路径。

英文摘要:

Learning-based methods have improved locomotion skills of quadruped robots through deep reinforcement learning. However, the sim-to-real gap and low sample efficiency still limit the skill transfer. To address this issue, we propose an efficient model-based learning framework that combines a world model with a policy network. We train a differentiable world model to predict future states and use it to directly supervise a Variational Autoencoder (VAE)-based policy network to imitate real animal behaviors. This significantly reduces the need for real interaction data and allows for rapid policy updates. We also develop a high-level network to track diverse commands and trajectories. Our simulated results show a tenfold sample efficiency increase compared to reinforcement learning methods such as PPO. In real-world testing, our policy achieves proficient command-following performance with only a two-minute data collection period and generalizes well to new speeds and paths.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑