arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.11779cs.LGstat.ML
收藏

深度强化学习的强化算法

Emphatic Algorithms for Deep Reinforcement Learning

DeepMind(深度思维);Amii(阿尔伯塔机器智能研究所);University of Alberta(阿尔伯塔大学)

Ray Jiang, Tom Zahavy, Zhongwen Xu, Adam White, Matteo Hessel, Charles Blundell, Hado van Hasselt

摘要:

离策略学习允许我们从不同行为策略产生的经验中学习可能的行为策略。时间差分(TD)学习算法在与函数近似和偏离策略采样相结合时可能会变得不稳定-这被称为“致命三元组”。强调时间差(ETD($λ$))算法通过适当地加权TD($λ$)更新来确保线性情况下的收敛。在本文中,我们将学习方法的使用扩展到深度强化学习代理。我们表明,天真地将ETD($λ$)适应流行的深度强化学习算法,该算法使用前向视图多步返回,导致性能不佳。然后,我们推导出新的迭代算法中使用的上下文中,这样的算法,我们证明,他们提供了显着的好处,在小问题,旨在突出TD方法的不稳定性。最后,我们在Arcade Learning Environment的经典Atari游戏上大规模应用这些算法时,观察到了性能的提高。

英文摘要:

Off-policy learning allows us to learn about possible policies of behavior from experience generated by a different behavior policy. Temporal difference (TD) learning algorithms can become unstable when combined with function approximation and off-policy sampling - this is known as the ''deadly triad''. Emphatic temporal difference (ETD($λ$)) algorithm ensures convergence in the linear case by appropriately weighting the TD($λ$) updates. In this paper, we extend the use of emphatic methods to deep reinforcement learning agents. We show that naively adapting ETD($λ$) to popular deep reinforcement learning algorithms, which use forward view multi-step returns, results in poor performance. We then derive new emphatic algorithms for use in the context of such algorithms, and we demonstrate that they provide noticeable benefits in small problems designed to highlight the instability of TD methods. Finally, we observed improved performance when applying these algorithms at scale on classic Atari games from the Arcade Learning Environment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑