Elastic Step DQN:一种减轻深度QNetworks中高估的新型多步算法
Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworks
摘要:
Deep Q-Networks算法(DQN)是第一个使用深度神经网络的强化学习算法,在许多Atari学习环境中成功超越人类水平。然而,发散和不稳定的行为一直是DQN长期存在的问题。不稳定行为通常以$Q$值的高估为特征,通常称为高估偏倚。为了解决高估偏差和发散行为,已经提出了一些启发式扩展。值得注意的是,多步更新已被证明可以大大减少不稳定的行为,同时提高代理的训练性能。然而,代理通常对多步更新范围($n$)的选择非常敏感,我们的经验实验表明,$n$的静态值选择不当在许多情况下会导致比单步DQN更差的性能。受$n$ -step DQN的成功和多步更新对高估偏差的影响的启发,本文提出了一种新的算法,我们称之为“弹性步长DQN”(ES-DQN)。它动态地改变步长的水平在多步更新的基础上访问的状态的相似性。我们的实证评估表明,ES-DQN在几个OpenAI Gym环境中的性能优于固定$n$更新的$n$ -step,Double DQN和Average DQN,同时减轻了高估偏差。
英文摘要:
Deep Q-Networks algorithm (DQN) was the first reinforcement learning algorithm using deep neural network to successfully surpass human level performance in a number of Atari learning environments. However, divergent and unstable behaviour have been long standing issues in DQNs. The unstable behaviour is often characterised by overestimation in the $Q$-values, commonly referred to as the overestimation bias. To address the overestimation bias and the divergent behaviour, a number of heuristic extensions have been proposed. Notably, multi-step updates have been shown to drastically reduce unstable behaviour while improving agent's training performance. However, agents are often highly sensitive to the selection of the multi-step update horizon ($n$), and our empirical experiments show that a poorly chosen static value for $n$ can in many cases lead to worse performance than single-step DQN. Inspired by the success of $n$-step DQN and the effects that multi-step updates have on overestimation bias, this paper proposes a new algorithm that we call `Elastic Step DQN' (ES-DQN). It dynamically varies the step size horizon in multi-step updates based on the similarity of states visited. Our empirical evaluation shows that ES-DQN out-performs $n$-step with fixed $n$ updates, Double DQN and Average DQN in several OpenAI Gym environments while at the same time alleviating the overestimation bias.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。