释放模拟器的潜力:设计时考虑RL
Unlocking the Potential of Simulators: Design with RL in Mind
摘要:
在仿真中使用强化学习(RL)来构建在现实生活中有用的策略是具有挑战性的。这通常归因于顺序决策方面:模拟中的不准确性在多个步骤中积累,因此模拟的轨迹与现实中发生的情况不同。在我们的工作中,我们需要考虑另一个重要的方面:在模拟控制的不匹配。我们注意到需要建模控制以及动态,因为过度简化的假设应用RL政策的行动可能会使政策在现实世界的系统失败。我们设计了一个模拟器来解决一个旋转任务(在机器人技术中感兴趣的),并证明,即使是一个简单的模拟器设计与RL铭记优于高保真模拟器,当涉及到学习的政策,是部署在一个真正的机器人系统。我们表明,一个现象,很难建模-摩擦-可以成功地利用,即使当RL是使用一个简单的动态和噪声模型的模拟器进行。因此,我们证明,只要确定了不确定性的主要来源,即使使用简单的模拟器,也可以学习适用于真实系统的策略。RL兼容的模拟器可以为在各个领域应用广泛的RL算法提供可能性。这一点很重要,因为目前医疗保健和教育等领域的数据稀疏性经常迫使研究人员和工程师只考虑样本高效的RL方法。成功的仿真器辅助强化学习可以增加实验强化学习算法的灵活性,并有助于将强化学习策略应用于数据稀缺领域的现实环境。我们相信,在机器人学的经验教训可以帮助其他领域设计RL兼容的模拟器,所以我们总结我们的经验,并提出建议。
英文摘要:
Using Reinforcement Learning (RL) in simulation to construct policies useful in real life is challenging. This is often attributed to the sequential decision making aspect: inaccuracies in simulation accumulate over multiple steps, hence the simulated trajectories diverge from what would happen in reality. In our work we show the need to consider another important aspect: the mismatch in simulating control. We bring attention to the need for modeling control as well as dynamics, since oversimplifying assumptions about applying actions of RL policies could make the policies fail on real-world systems. We design a simulator for solving a pivoting task (of interest in Robotics) and demonstrate that even a simple simulator designed with RL in mind outperforms high-fidelity simulators when it comes to learning a policy that is to be deployed on a real robotic system. We show that a phenomenon that is hard to model - friction - could be exploited successfully, even when RL is performed using a simulator with a simple dynamics and noise model. Hence, we demonstrate that as long as the main sources of uncertainty are identified, it could be possible to learn policies applicable to real systems even using a simple simulator. RL-compatible simulators could open the possibilities for applying a wide range of RL algorithms in various fields. This is important, since currently data sparsity in fields like healthcare and education frequently forces researchers and engineers to only consider sample-efficient RL approaches. Successful simulator-aided RL could increase flexibility of experimenting with RL algorithms and help applying RL policies to real-world settings in fields where data is scarce. We believe that lessons learned in Robotics could help other fields design RL-compatible simulators, so we summarize our experience and conclude with suggestions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。