arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.04017cs.LGcs.AImath.OC
收藏

深度神经网络时序差分学习的改进有限时间分析

An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks

Center for Data Science, Peking University(北京大学数据科学中心);Beijing International Center for Mathematical Research(北京国际数学研究中心);Center for Machine Learning Research(机器学习研究中心);Changsha Institute for Computing and Digital Economy(长沙计算与数字经济研究院);National University of Singapore(新加坡国立大学)

Zhifa Ke, Zaiwen Wen, Junyu Zhang

AI总结:

本文研究神经网络时序差分学习的有限时间收敛理论,通过新的证明技术将马尔可夫采样下的样本复杂度从 $\tilde{\mathcal{O}}(ε^{-2})$ 改进至 $\tilde{\mathcal{O}}(ε^{-1})$。

摘要:

采用神经网络进行函数参数化的时序差分(TD)学习算法,已在许多实际大规模强化学习任务中取得充分验证的经验成功。然而,由于动作价值近似的非线性,对这些算法的理论理解仍然具有挑战性。本文针对使用一般 $L$ 层神经网络的神经 TD 方法,发展了一种改进的非渐近分析。我们提出了新的证明技术,并推导出改进的新样本复杂度 $\tilde{\mathcal{O}}(ε^{-1})$。据我们所知,这是首次在马尔可夫采样下实现 $\tilde{\mathcal{O}}(ε^{-1})$ 复杂度的神经 TD 有限时间分析,而现有文献中最知名的复杂度为 $\tilde{\mathcal{O}}(ε^{-2})$。

英文摘要:

Temporal difference (TD) learning algorithms with neural network function parameterization have well-established empirical success in many practical large-scale reinforcement learning tasks. However, theoretical understanding of these algorithms remains challenging due to the nonlinearity of the action-value approximation. In this paper, we develop an improved non-asymptotic analysis of the neural TD method with a general $L$-layer neural network. New proof techniques are developed and an improved new $\tilde{\mathcal{O}}(ε^{-1})$ sample complexity is derived. To our best knowledge, this is the first finite-time analysis of neural TD that achieves an $\tilde{\mathcal{O}}(ε^{-1})$ complexity under the Markovian sampling, as opposed to the best known $\tilde{\mathcal{O}}(ε^{-2})$ complexity in the existing literature.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑