arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.09010cs.ROcs.LGcs.SYeess.SY
收藏

DiAReL:面向机器人控制中鲁棒Sim2Real策略迁移的具备扰动感知的强化学习方法

DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control

Department of Computer Engineering, School of Computation, Information and Technology, Technical University of Munich(计算机工程系,计算、信息与技术学院,慕尼黑技术大学);Rosenheim University of Applied Sciences(罗森海姆应用技术大学)

Mohammadhossein Malmir, Josip Josifovski, Noah Klarmann, Alois Knoll

AI总结:

针对机器人控制中仿真模型近似导致sim2real迁移受阻的问题,提出带扰动感知的强化学习方法DiAReL,引入DAMDP将扰动估计融入同策略RL训练,提升控制鲁棒性与迁移成功率。

摘要:

延迟马尔可夫决策过程(DMDP)通过为智能体的状态空间扩充有限时间窗口内近期执行的动作来满足马尔可夫性质。依托这类状态扩充,延迟求解强化学习算法训练策略,以学习与存在观测或动作延迟的环境的最优交互方式。尽管这类方法可直接在真实机器人上训练,但受样本效率低、资源有限或安全约束影响,常见做法是将仿真中训练的模型迁移到实体机器人上。然而机器人仿真依赖物理系统的近似模型,这会阻碍sim2real(仿真到真实)迁移。本研究将机器人或环境动力学建模中的各类不确定性视为施加于系统输入的未知内在扰动,在延迟场景下提出扰动增强马尔可夫决策过程(DAMDP)作为全新表示形式,将扰动估计融入同策略强化学习算法的训练过程。所提方法在机器人到达与推动任务的学习中通过多项指标完成验证,并与无扰动感知的基线方法对比。结果显示,扰动增强模型可在控制响应中实现更高的稳定性与鲁棒性,进而提升sim2real成功迁移的可能性。

英文摘要:

Delayed Markov decision processes (DMDPs) fulfill the Markov property by augmenting the state space of agents with a finite time window of recently committed actions. In reliance on these state augmentations, delay-resolved reinforcement learning algorithms train policies to learn optimal interactions with environments featuring observation or action delays. Although such methods can be directly trained on the real robots, due to sample inefficiency, limited resources, or safety constraints, a common approach is to transfer models trained in simulation to the physical robot. However, robotic simulations rely on approximated models of the physical systems, which hinders the sim2real transfer. In this work, we consider various uncertainties in modeling the robot or environment dynamics as unknown intrinsic disturbances applied to the system input. We introduce the disturbance-augmented Markov decision process (DAMDP) in delayed settings as a novel representation to incorporate disturbance estimation in training on-policy reinforcement learning algorithms. The proposed method is validated across several metrics on learning robotic reaching and pushing tasks and compared with disturbance-unaware baselines. The results show that the disturbance-augmented models can achieve higher stabilization and robustness in the control response, which in turn improves the prospects of successful sim2real transfer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑