arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.02635cs.ROcs.AIcs.LG
收藏

基于基础先验的强化学习:让具身智能体高效自主学习

Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own

Tsinghua University(清华大学);Shanghai Qi Zhi Institute(上海期智研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);UC Berkeley(加州大学伯克利分校)

Weirui Ye, Yunsheng Zhang, Haoyang Weng, Xianfan Gu, Shengjie Wang, Tong Zhang, Mengchen Wang, Pieter Abbeel, Yang Gao

AI总结:

本文提出基于基础先验的强化学习框架RLFP及FAC算法,利用基础模型提供自动奖励与指导,使具身智能体在真实和模拟操作任务中实现高效自主学习并取得高成功率。

摘要:

强化学习(RL)是解决机器人操作任务的一种有前景的方法。然而,直接在现实世界中应用RL算法具有挑战性。一方面,RL是数据密集型的,通常需要与环境进行数百万次交互,这在现实场景中是不切实际的。另一方面,需要投入大量工程精力手动设计奖励函数。为了解决这些问题,本文利用了基础模型。我们提出了基于基础先验的强化学习(RLFP),以利用来自策略、价值和成功奖励基础模型的指导和反馈。在此框架内,我们引入了基础模型引导的Actor-Critic(FAC)算法,该算法使具身智能体能够利用自动奖励函数更高效地探索。我们框架的优势有三方面:(1)样本高效;(2)最小且有效的奖励工程;(3)对基础模型形式不可知且对噪声先验具有鲁棒性。我们的方法在真实机器人和模拟环境的各种操作任务中均取得了显著性能。在5个真实机器人灵巧任务中,FAC在一小时实时学习后平均成功率达到86%。在模拟的Meta-world的8个任务中,FAC在不到100k帧(约1小时训练)的情况下,在7/8的任务中达到100%的成功率,优于在1M帧下使用手动设计奖励的基线方法。我们相信RLFP框架能够使未来的机器人在物理世界中自主探索和学习更多任务。可视化与代码可在https://yewr.github.io/rlfp获取。

英文摘要:

Reinforcement learning (RL) is a promising approach for solving robotic manipulation tasks. However, it is challenging to apply the RL algorithms directly in the real world. For one thing, RL is data-intensive and typically requires millions of interactions with environments, which are impractical in real scenarios. For another, it is necessary to make heavy engineering efforts to design reward functions manually. To address these issues, we leverage foundation models in this paper. We propose Reinforcement Learning with Foundation Priors (RLFP) to utilize guidance and feedback from policy, value, and success-reward foundation models. Within this framework, we introduce the Foundation-guided Actor-Critic (FAC) algorithm, which enables embodied agents to explore more efficiently with automatic reward functions. The benefits of our framework are threefold: (1) \textit{sample efficient}; (2) \textit{minimal and effective reward engineering}; (3) \textit{agnostic to foundation model forms and robust to noisy priors}. Our method achieves remarkable performances in various manipulation tasks on both real robots and in simulation. Across 5 dexterous tasks with real robots, FAC achieves an average success rate of 86\% after one hour of real-time learning. Across 8 tasks in the simulated Meta-world, FAC achieves 100\% success rates in 7/8 tasks under less than 100k frames (about 1-hour training), outperforming baseline methods with manual-designed rewards in 1M frames. We believe the RLFP framework can enable future robots to explore and learn autonomously in the physical world for more tasks. Visualizations and code are available at \url{https://yewr.github.io/rlfp}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑