arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1901.07510cs.LGstat.ML
收藏

理解多步深度强化学习:对DQN目标的系统研究

Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target

University of Alberta(阿尔伯塔大学)

J. Fernando Hernandez-Garcia, Richard S. Sutton

摘要:

多步方法,如Retrace($λ$)和$n$ -步$Q$ -学习已经成为现代深度强化学习代理的重要组成部分。这些方法通常作为更大架构的一部分进行评估,并且它们的评估很少包括足够的样本来得出关于其性能的统计上显著的结论。这种类型的方法很难理解多步方法的特定算法细节如何影响学习。在本文中,我们结合了$n$ -步骤动作值算法Retrace,$Q$ -学习,树备份,Sarsa,和$Q(σ)$与类似于DQN的架构。我们在山地车环境中测试了所有这些算法的性能;这种环境的选择允许更快的训练时间和更大的样本量。我们提出了对这些算法的性能的影响的政策外校正,备份长度参数$n$,和目标网络的更新频率的统计分析。我们的研究结果表明:(1)使用非策略校正会对Sarsa和$Q(σ)$的性能产生不利影响;(2)增加备份长度$n$在所有不同的算法中一致地提高了性能;以及(3)Sarsa和$Q$学习的性能比Tree Backup、$Q(σ)$、在这个特殊的任务中。

英文摘要:

Multi-step methods such as Retrace($λ$) and $n$-step $Q$-learning have become a crucial component of modern deep reinforcement learning agents. These methods are often evaluated as a part of bigger architectures and their evaluations rarely include enough samples to draw statistically significant conclusions about their performance. This type of methodology makes it difficult to understand how particular algorithmic details of multi-step methods influence learning. In this paper we combine the $n$-step action-value algorithms Retrace, $Q$-learning, Tree Backup, Sarsa, and $Q(σ)$ with an architecture analogous to DQN. We test the performance of all these algorithms in the mountain car environment; this choice of environment allows for faster training times and larger sample sizes. We present statistical analyses on the effects of the off-policy correction, the backup length parameter $n$, and the update frequency of the target network on the performance of these algorithms. Our results show that (1) using off-policy correction can have an adverse effect on the performance of Sarsa and $Q(σ)$; (2) increasing the backup length $n$ consistently improved performance across all the different algorithms; and (3) the performance of Sarsa and $Q$-learning was more robust to the effect of the target network update frequency than the performance of Tree Backup, $Q(σ)$, and Retrace in this particular task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑