arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.18518cs.RO
收藏

基于在线学习的机器人控制高效基于模型的强化学习

Efficient Model-Based Reinforcement Learning for Robot Control via Online Learning

CREATE Lab, EPFL, Lausanne, Switzerland(洛桑联邦理工学院CREATE实验室)

Fang Nan, Hao Ma, Qinghua Guan, Josie Hughes, Michael Muehlebach, Marco Hutter

AI总结:

本研究提出一种在线基于模型的强化学习算法,基于实时交互数据构建动力学模型并更新策略,在液压挖掘机臂和软体机器人臂上验证了其高样本效率与鲁棒自适应能力,为机器人本体高效学习提供了新方案。

摘要:

我们提出了一种适用于直接在现实世界中控制复杂机器人系统的在线基于模型的强化学习算法。与依赖大量离线仿真和无模型策略优化的主流sim-to-real(仿真到现实)流程不同,我们的方法从实时交互数据中构建动力学模型,并在学习到的动力学模型引导下执行策略更新。这种高效的基于模型的强化学习方案显著减少了训练控制策略所需的样本量,能够直接基于现实世界的 rollout(滚动执行)数据进行训练,大幅降低了仿真数据偏差的影响,便于搜索高性能控制策略。我们采用在线学习分析,在标准随机在线优化假设下推导出次线性遗憾界,为随着交互数据增多而提升的性能提供了形式化保证。我们在液压挖掘机臂和软体机器人臂上开展了实验评估,该算法相比无模型强化学习方法展现出极强的样本效率,数小时内即可达到相当的性能;在载荷条件随机化时,还观测到其对动态变化的鲁棒自适应能力。我们的方法为面向各类高难度控制任务的高效可靠机器人本体学习铺平了道路。

英文摘要:

We present an online model-based reinforcement learning algorithm suitable for controlling complex robotic systems directly in the real world. Unlike prevailing sim-to-real pipelines that rely on extensive offline simulation and model-free policy optimization, our method builds a dynamics model from real-time interaction data and performs policy updates guided by the learned dynamics model. This efficient model-based reinforcement learning scheme significantly reduces the number of samples to train control policies, enabling direct training on real-world rollout data. This significantly reduces the influence of bias in the simulated data, and facilitates the search for high-performance control policies. We adopt online learning analysis to derive sublinear regret bounds under standard stochastic online optimization assumptions, providing formal guarantees on performance improvement as more interaction data are collected. Experimental evaluations were performed on a hydraulic excavator arm and a soft robot arm, where the algorithm demonstrates strong sample efficiency compared to model-free reinforcement learning methods, reaching comparable performance within hours. Robust adaptation to shifting dynamics was also observed when the payload condition was randomized. Our approach paves the way toward efficient and reliable on-robot learning for a broad class of challenging control tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑