缩小多机器人协作深度强化学习中的虚实差距
Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning
摘要:
目前深度强化学习的研究方向包括弥合模拟-现实差距,提高分布式多智能体强化学习中经验的样本效率,以及开发分布式学习中对抗对抗性智能体的鲁棒方法等。在这项工作中,我们特别感兴趣的是分析如何多智能体强化学习可以弥合差距,在分布式多机器人系统的不同机器人的操作不一定是同质的现实。这些变化可能是由于感测失配、机械接头校准方面的固有误差或精度的简单差异而发生的。虽然我们的结果是基于模拟的,但我们引入了分布式强化学习中的传感,校准和准确性失配的影响与近端策略优化(PPO)。我们讨论了如何不同类型的扰动,以及如何代理的数量经历这些扰动影响协作学习的努力。模拟使用Bullet物理引擎中的Kuka手臂模型进行。这是,据我们所知,第一次工作,探讨PPO在多机器人系统的限制时,考虑到不同的机器人可能会暴露在不同的环境中,他们的传感器或执行器引起的错误。随着这项工作的结论,我们为未来的工作设计和开发方法,以实现强大的强化学习的存在下,现实世界的扰动,可能会有所不同,在一个多机器人系统的起点。
英文摘要:
Current research directions in deep reinforcement learning include bridging the simulation-reality gap, improving sample efficiency of experiences in distributed multi-agent reinforcement learning, together with the development of robust methods against adversarial agents in distributed learning, among many others. In this work, we are particularly interested in analyzing how multi-agent reinforcement learning can bridge the gap to reality in distributed multi-robot systems where the operation of the different robots is not necessarily homogeneous. These variations can happen due to sensing mismatches, inherent errors in terms of calibration of the mechanical joints, or simple differences in accuracy. While our results are simulation-based, we introduce the effect of sensing, calibration, and accuracy mismatches in distributed reinforcement learning with proximal policy optimization (PPO). We discuss on how both the different types of perturbances and how the number of agents experiencing those perturbances affect the collaborative learning effort. The simulations are carried out using a Kuka arm model in the Bullet physics engine. This is, to the best of our knowledge, the first work exploring the limitations of PPO in multi-robot systems when considering that different robots might be exposed to different environments where their sensors or actuators have induced errors. With the conclusions of this work, we set the initial point for future work on designing and developing methods to achieve robust reinforcement learning on the presence of real-world perturbances that might differ within a multi-robot system.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。