arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.08583math.OCcs.AI
收藏

Q-learning的离散时间切换系统分析

A Discrete-Time Switching System Analysis of Q-learning

Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院(KAIST));Purdue University(普渡大学);ETH Zürich(苏黎世联邦理工学院)

Donghwan Lee, Jianghai Hu, Niao He

AI总结:

本文提出基于控制理论的离散时间随机仿射切换系统框架,分析恒定步长异步Q-learning的非渐近收敛性,推导出新的有限时间误差界限,并揭示了其高估现象。

摘要:

本文开发了一种新颖的控制理论框架来分析Q-learning的非渐近收敛性。我们证明了采用恒定步长的异步Q-learning的动态可以自然地表述为离散时间随机仿射切换系统。此外,Q-learning估计误差的演化由两个较简单动态系统的轨迹进行高估和低估。基于这两个系统,我们推导出了使用恒定步长时异步Q-learning的新的有限时间误差界限。我们的分析还揭示了Q-learning的高估现象。我们进一步通过数值仿真说明并验证了该分析。

英文摘要:

This paper develops a novel control-theoretic framework to analyze the non-asymptotic convergence of Q-learning. We show that the dynamics of asynchronous Q-learning with a constant step-size can be naturally formulated as a discrete-time stochastic affine switching system. Moreover, the evolution of the Q-learning estimation error is over- and underestimated by trajectories of two simpler dynamical systems. Based on these two systems, we derive a new finite-time error bound of asynchronous Q-learning when a constant stepsize is used. Our analysis also sheds light on the overestimation phenomenon of Q-learning. We further illustrate and validate the analysis through numerical simulations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑