arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1912.04511cs.LGmath.OCstat.ML
收藏

基于神经网络函数逼近的Q学习算法的实时分析

A Finite-Time Analysis of Q-Learning with Neural Network Function Approximation

University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

Pan Xu, Quanquan Gu

摘要:

具有神经网络函数逼近的Q学习(简称神经Q学习)是最流行的深度强化学习算法之一。尽管它的经验成功,神经Q学习的非渐近收敛速度仍然几乎是未知的。在本文中,我们提出了一种神经Q学习算法的有限时间分析,其中数据是从马尔可夫决策过程中生成的,并且动作价值函数由深度ReLU神经网络近似。我们证明了,神经Q学习找到最优的政策与$O(1/\sqrt{T})$收敛速度,如果神经函数逼近充分过参数化,其中$T$是迭代次数。据我们所知,我们的结果是第一个有限时间分析下的非独立同分布神经Q学习。数据假设

英文摘要:

Q-learning with neural network function approximation (neural Q-learning for short) is among the most prevalent deep reinforcement learning algorithms. Despite its empirical success, the non-asymptotic convergence rate of neural Q-learning remains virtually unknown. In this paper, we present a finite-time analysis of a neural Q-learning algorithm, where the data are generated from a Markov decision process and the action-value function is approximated by a deep ReLU neural network. We prove that neural Q-learning finds the optimal policy with $O(1/\sqrt{T})$ convergence rate if the neural function approximator is sufficiently overparameterized, where $T$ is the number of iterations. To our best knowledge, our result is the first finite-time analysis of neural Q-learning under non-i.i.d. data assumption.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑