arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1903.08894cs.LGcs.AI
收藏

在深度Q学习中描述发散性

Towards Characterizing Divergence in Deep Q-Learning

OpenAI;UC Berkeley(加州大学伯克利分校);The Nueva School(新学院)

Joshua Achiam, Ethan Knight, Pieter Abbeel

摘要:

深度Q学习(DQL)是一种用于控制的时间差分算法,它采用了三种技术,统称为强化学习中的“致命三元组”:自举、离策略学习和函数逼近。先前的工作已经证明,这些一起可以导致Q-学习算法的发散,但发散发生的条件还没有得到很好的理解。在这篇文章中,我们给出了一个简单的分析的基础上的线性近似的Q值更新,我们相信这提供了深入了解下的致命三位一体的分歧。我们分析的中心点是考虑深度Q更新的前导阶近似何时是或不是超范数的收缩。基于这种分析,我们开发了一种算法,该算法允许稳定的深度Q学习用于连续控制,而无需任何常规使用的技巧(例如目标网络,自适应梯度优化器或使用多个Q函数)。我们证明了我们的算法在OpenAI Gym的标准MuJoCo基准测试中的性能高于或接近最先进的水平。

英文摘要:

Deep Q-Learning (DQL), a family of temporal difference algorithms for control, employs three techniques collectively known as the `deadly triad' in reinforcement learning: bootstrapping, off-policy learning, and function approximation. Prior work has demonstrated that together these can lead to divergence in Q-learning algorithms, but the conditions under which divergence occurs are not well-understood. In this note, we give a simple analysis based on a linear approximation to the Q-value updates, which we believe provides insight into divergence under the deadly triad. The central point in our analysis is to consider when the leading order approximation to the deep-Q update is or is not a contraction in the sup norm. Based on this analysis, we develop an algorithm which permits stable deep Q-learning for continuous control without any of the tricks conventionally used (such as target networks, adaptive gradient optimizers, or using multiple Q functions). We demonstrate that our algorithm performs above or near state-of-the-art on standard MuJoCo benchmarks from the OpenAI Gym.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑