arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2002.09795cs.LGmath.OCstat.ML
收藏

周期性Q学习

Periodic Q-Learning

Korea Advanced Institute of Science and Technology(韩国科学技术院);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Donghwan Lee, Niao He

摘要:

使用目标网络是深度强化学习中稳定训练的常见做法;然而,对这种技术的理论理解仍然有限。在本文中,我们研究了所谓的周期性Q-学习算法(简称PQ-学习),它类似于深度Q-学习中用于解决表格设置中的无限时域折扣马尔可夫决策过程(DMDP)的技术。PQ学习维护两个独立的Q值估计-在线估计和目标估计。在线估计遵循标准Q学习更新,而目标估计定期更新。与标准Q学习相比,PQ学习具有简单的有限时间分析,并且在寻找ε最优策略时实现了更好的样本复杂度。我们的结果提供了一个初步的理由,利用目标估计或网络的有效性Q-学习算法。

英文摘要:

The use of target networks is a common practice in deep reinforcement learning for stabilizing the training; however, theoretical understanding of this technique is still limited. In this paper, we study the so-called periodic Q-learning algorithm (PQ-learning for short), which resembles the technique used in deep Q-learning for solving infinite-horizon discounted Markov decision processes (DMDP) in the tabular setting. PQ-learning maintains two separate Q-value estimates - the online estimate and target estimate. The online estimate follows the standard Q-learning update, while the target estimate is updated periodically. In contrast to the standard Q-learning, PQ-learning enjoys a simple finite time analysis and achieves better sample complexity for finding an epsilon-optimal policy. Our result provides a preliminary justification of the effectiveness of utilizing target estimates or networks in Q-learning algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑