离散和连续动作任务的候选动作驱动裁剪双Q学习
Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Tasks
摘要：
双Q学习是马尔可夫决策过程（MDP）问题中一种流行的强化学习算法。Clipped Double Q-learning是Double Q-learning的一个有效变体，它使用Clipped Double Estimator来近似最大期望动作值。在某些随机环境中，由于截断双Q估计器的低估偏差，截断双Q学习的性能可能会下降。为了减小低估偏差，本文提出了一种基于动作候选的双Q学习裁剪双估计器。具体来说，我们首先从一组估计量中选择一组具有高动作值的精英动作候选者。然后，在这些候选者中，我们从另一组估计量中选择最高值的动作。最后，我们使用第二组估计器中的最大值来裁剪第一组估计器中所选动作的动作值，并且裁剪的值用于近似最大期望动作值。从理论上讲，我们的裁剪双Q学习中的低估偏差随着候选动作数量的减少而单调衰减。此外，候选行动的数量控制高估和低估偏差之间的权衡。此外，我们还通过近似精英连续动作候选人，将我们的裁剪双Q学习扩展到连续动作任务。我们的经验验证，我们的算法可以更准确地估计一些玩具环境中的最大预期动作值，并在几个基准问题产生良好的性能。
英文摘要：
Double Q-learning is a popular reinforcement learning algorithm in Markov decision process (MDP) problems. Clipped Double Q-learning, as an effective variant of Double Q-learning, employs the clipped double estimator to approximate the maximum expected action value. Due to the underestimation bias of the clipped double estimator, the performance of clipped Double Q-learning may be degraded in some stochastic environments. In this paper, in order to reduce the underestimation bias, we propose an action candidate-based clipped double estimator for Double Q-learning. Specifically, we first select a set of elite action candidates with high action values from one set of estimators. Then, among these candidates, we choose the highest valued action from the other set of estimators. Finally, we use the maximum value in the second set of estimators to clip the action value of the chosen action in the first set of estimators and the clipped value is used for approximating the maximum expected action value. Theoretically, the underestimation bias in our clipped Double Q-learning decays monotonically as the number of action candidates decreases. Moreover, the number of action candidates controls the trade-off between the overestimation and underestimation biases. In addition, we also extend our clipped Double Q-learning to continuous action tasks via approximating the elite continuous action candidates. We empirically verify that our algorithm can more accurately estimate the maximum expected action value on some toy environments and yield good performance on several benchmark problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
