深度强化学习和致命的三合一
Deep Reinforcement Learning and the Deadly Triad
摘要:
我们从强化学习理论中知道,时间差学习在某些情况下可能会失败。Sutton和Barto(2018)确定了函数近似、自举和离策略学习的致命三位一体。当这三个属性结合在一起时,学习可以随着价值估计变得无界而发散。然而,一些算法成功地结合了这三个属性,这表明我们的理解至少存在部分差距。在这项工作中,我们调查了致命的三合会在实践中的影响,在一系列流行的深度强化学习模型的背景下-用经验重放训练的深度Q网络-分析这个系统的组件如何在致命三合会的出现中发挥作用,以及代理的表现
英文摘要:
We know from reinforcement learning theory that temporal difference learning can fail in certain cases. Sutton and Barto (2018) identify a deadly triad of function approximation, bootstrapping, and off-policy learning. When these three properties are combined, learning can diverge with the value estimates becoming unbounded. However, several algorithms successfully combine these three properties, which indicates that there is at least a partial gap in our understanding. In this work, we investigate the impact of the deadly triad in practice, in the context of a family of popular deep reinforcement learning models - deep Q-networks trained with experience replay - analysing how the components of this system play a role in the emergence of the deadly triad, and in the agent's performance
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。