arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.02632cs.LGcs.SYeess.SYmath.OC
收藏

通过设计与乐观性实现Q-Learning的稳定性

Stability of Q-Learning Through Design and Optimism

University of Florida(佛罗里达大学)

Sean Meyn

AI总结:

本文既是随机逼近与Q-learning的教程,也提出两项新成果:通过改进Gibbs策略的乐观训练证明带线性函数逼近的Q-learning稳定,以及提出无需矩阵求逆的Zap Zero算法,可用于含非线性函数逼近的Q-learning等随机逼近场景。

摘要:

自20世纪80年代Chris Watkins在其博士论文中提出Q-learning以来,它已成为强化学习工具包的重要组成部分。本文部分内容是关于随机逼近与Q-learning的教程,详细介绍了2023年6月在法国南锡举行的INFORMS APS首届应用概率信托全会讲座的相关内容。\n本文还提出了确保这些算法以及其他场景下随机逼近方法稳定性、并有望加速收敛的新方法。其中包含两项全新的贡献:\n1. 带线性函数逼近的Q-learning的稳定性是三十多年来一直悬而未决的研究课题。本文证明,通过以改进的Gibbs策略形式进行适当的乐观训练,投影Bellman方程存在解,且算法是稳定的(即参数估计有界)。收敛性仍是众多未解决的研究课题之一。\n2. 全新的Zap Zero算法旨在无需矩阵求逆即可近似Newton-Raphson流。在对算法的平均流向量场的温和假设、以及对底层马尔可夫链的相容性统计假设下,该算法稳定且收敛。该算法是一种通用的随机逼近方法,尤其适用于采用“盲”训练的Q-learning,即便使用非线性函数逼近也同样适用。

英文摘要:

Q-learning has become an important part of the reinforcement learning toolkit since its introduction in the dissertation of Chris Watkins in the 1980s. The purpose of this paper is in part a tutorial on stochastic approximation and Q-learning, providing details regarding the INFORMS APS inaugural Applied Probability Trust Plenary Lecture, presented in Nancy France, June 2023. The paper also presents new approaches to ensure stability and potentially accelerated convergence for these algorithms, and stochastic approximation in other settings. Two contributions are entirely new: 1. Stability of Q-learning with linear function approximation has been an open topic for research for over three decades. It is shown that with appropriate optimistic training in the form of a modified Gibbs policy, there exists a solution to the projected Bellman equation, and the algorithm is stable (in terms of bounded parameter estimates). Convergence remains one of many open topics for research. 2. The new Zap Zero algorithm is designed to approximate the Newton-Raphson flow without matrix inversion. It is stable and convergent under mild assumptions on the mean flow vector field for the algorithm, and compatible statistical assumption on an underlying Markov chain. The algorithm is a general approach to stochastic approximation which in particular applies to Q-learning with "oblivious" training even with non-linear function approximation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑