UCB动量Q学习：在不遗忘的情况下纠正偏差
UCB Momentum Q-learning: Correcting the bias without forgetting
摘要：
我们提出了UCBMQ，上置信界动量Q-学习，一种新的算法，用于表格和可能的阶段依赖性，情节马尔可夫决策过程的强化学习。UCBMQ是基于Q学习的，我们在Q学习中加入了动量项，并依靠乐观原则来应对不确定性。UCBMQ的新技术成分是使用动量来纠正Q学习所遭受的偏见，同时限制它对后悔的二阶项的影响。对于UCBMQ，我们能够保证最多$O(\sqrt{H^3SAT}+ H^4 S A )$的遗憾，其中$H$是事件的长度，$S$是状态的数量，$A$是动作的数量，$T$是事件的数量，并且忽略poly- $\log(SAHT)$中的项。值得注意的是，UCBMQ是第一个同时匹配$Ω(\sqrt{H^3SAT})$的下限的算法，对于足够大的$T$，并且具有仅与状态数$S$线性缩放的二阶项（相对于水平$T$）。
英文摘要：
We propose UCBMQ, Upper Confidence Bound Momentum Q-learning, a new algorithm for reinforcement learning in tabular and possibly stage-dependent, episodic Markov decision process. UCBMQ is based on Q-learning where we add a momentum term and rely on the principle of optimism in face of uncertainty to deal with exploration. Our new technical ingredient of UCBMQ is the use of momentum to correct the bias that Q-learning suffers while, at the same time, limiting the impact it has on the second-order term of the regret. For UCBMQ, we are able to guarantee a regret of at most $O(\sqrt{H^3SAT}+ H^4 S A )$ where $H$ is the length of an episode, $S$ the number of states, $A$ the number of actions, $T$ the number of episodes and ignoring terms in poly-$\log(SAHT)$. Notably, UCBMQ is the first algorithm that simultaneously matches the lower bound of $Ω(\sqrt{H^3SAT})$ for large enough $T$ and has a second-order term (with respect to the horizon $T$) that scales only linearly with the number of states $S$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
