arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.02827cs.LG
收藏

Q学习几乎必然收敛的一个初等证明

An Elementary Proof that Q-learning Converges Almost Surely

University of Alberta(阿尔伯塔大学)

Matthew T. Regehr, Alex Ayoub

摘要:

Watkins和Dayan的Q学习是一种无模型强化学习算法,它通过随机“访问”许多状态对来迭代地改进MDP的最优动作价值函数的估计[Watkins和Dayan,1992]。该算法的变体是强化学习中许多最新成就的核心,包括超人Atari播放深度Q网络[Mnih等人,2015年]。本文的目标是重现一个精确且(几乎)独立的证明,证明Q学习收敛。许多现有的文献利用强大的理论,以获得高度概括的结果,在这方面。然而,这种方法要求读者熟悉不同的研究领域,并与之建立许多深刻的联系。一个学生试图加深他们对Q学习的理解,可能会陷入“RL学习地狱”的恶性循环。出于这个原因,我们给出了一个完整的证明,从开始到结束,只使用一个外部结果从随机逼近领域,尽管事实上,这种最小的依赖于其他结果是以牺牲一些“光泽”。

英文摘要:

Watkins' and Dayan's Q-learning is a model-free reinforcement learning algorithm that iteratively refines an estimate for the optimal action-value function of an MDP by stochastically "visiting" many state-ation pairs [Watkins and Dayan, 1992]. Variants of the algorithm lie at the heart of numerous recent state-of-the-art achievements in reinforcement learning, including the superhuman Atari-playing deep Q-network [Mnih et al., 2015]. The goal of this paper is to reproduce a precise and (nearly) self-contained proof that Q-learning converges. Much of the available literature leverages powerful theory to obtain highly generalizable results in this vein. However, this approach requires the reader to be familiar with and make many deep connections to different research areas. A student seeking to deepen their understand of Q-learning risks becoming caught in a vicious cycle of "RL-learning Hell". For this reason, we give a complete proof from start to finish using only one external result from the field of stochastic approximation, despite the fact that this minimal dependence on other results comes at the expense of some "shininess".

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑