arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1907.04685cs.ROcs.LG
收藏

强化学习从模拟到现实的可移植性评估

Assessing Transferability from Simulation to Reality for Reinforcement Learning

Technische Universität Darmstadt(达姆施塔特工业大学);Honda Research Institute Europe(本田欧洲研究院);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)

Fabio Muratore, Michael Gienger, Jan Peters

摘要:

从物理模拟中学习机器人控制策略对机器人社区非常感兴趣,因为它可以通过减轻对昂贵的真实世界实验的需求来使学习过程更快,更便宜和更安全。然而,将学到的行为从模拟直接转移到现实是一个重大挑战。在稍微有故障的模拟器上优化策略很容易导致“模拟优化偏差”(SOB)的最大化。在这种情况下,优化器利用模拟器的建模误差,使得所产生的行为可能会损坏机器人。我们通过应用域随机化来应对这一挑战,即,在学习期间随机化物理模拟的参数。我们提出了一种算法称为基于仿真的策略优化与可转移性评估(SPOTA),它使用的估计量的最大值,制定一个停止标准的训练。引入的估计量量化了训练时经历的域集合的过拟合。我们在两个不同的二阶非线性系统上的实验结果表明,新的基于仿真的策略搜索算法能够专门从随机模拟器中学习控制策略,该策略可以直接应用于实际系统而无需任何额外的训练。

英文摘要:

Learning robot control policies from physics simulations is of great interest to the robotics community as it may render the learning process faster, cheaper, and safer by alleviating the need for expensive real-world experiments. However, the direct transfer of learned behavior from simulation to reality is a major challenge. Optimizing a policy on a slightly faulty simulator can easily lead to the maximization of the `Simulation Optimization Bias` (SOB). In this case, the optimizer exploits modeling errors of the simulator such that the resulting behavior can potentially damage the robot. We tackle this challenge by applying domain randomization, i.e., randomizing the parameters of the physics simulations during learning. We propose an algorithm called Simulation-based Policy Optimization with Transferability Assessment (SPOTA) which uses an estimator of the SOB to formulate a stopping criterion for training. The introduced estimator quantifies the over-fitting to the set of domains experienced while training. Our experimental results on two different second order nonlinear systems show that the new simulation-based policy search algorithm is able to learn a control policy exclusively from a randomized simulator, which can be applied directly to real systems without any additional training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑