使用参考优势分解的 Q-learning 的间隙相关界
Gap-Dependent Bounds for Q-Learning using Reference-Advantage Decomposition
AI总结:
本文首次为采用方差估计器和参考优势分解的 Q-learning 算法(UCB-Advantage 和 Q-EarlySettled-Advantage)建立了对数级间隙相关遗憾界,并首次给出策略切换成本的间隙相关分析。
摘要:
我们研究有限时域回合制表格马尔可夫决策过程(MDPs)中两种重要的在线策略 Q-learning 算法的间隙相关界:UCB-Advantage(Zhang 等人,2020)和 Q-EarlySettled-Advantage(Li 等人,2021)。UCB-Advantage 和 Q-EarlySettled-Advantage 改进了基于 Hoeffding 型奖励的结果,并在最坏情况下实现了几乎最优的 $\sqrt{T}$ 型遗憾界,其中 $T$ 是总步数。然而,MDP 的良性结构(例如严格正的次优间隙)可以显著改善遗憾。虽然已经为使用 Hoeffding 型奖励的 Q-learning 获得了间隙相关遗憾界,但对于在其奖励中使用方差估计器并采用参考优势分解进行方差缩减的 Q-learning,建立间隙相关遗憾界仍然是一个悬而未决的问题。我们开发了一种新颖的误差分解框架,以证明 UCB-Advantage 和 Q-EarlySettled-Advantage 的间隙相关遗憾界,这些界关于 $T$ 是对数级的,并且优于现有的 Q-learning 算法的界。此外,我们建立了 UCB-Advantage 的策略切换成本的间隙相关界,并在最坏情况 MDP 下改进了该界。据我们所知,本文首次对使用方差估计器和参考优势分解的 Q-learning 进行了间隙相关遗憾分析,并且首次对 Q-learning 的策略切换成本进行了间隙相关分析。
英文摘要:
We study the gap-dependent bounds of two important algorithms for on-policy Q-learning for finite-horizon episodic tabular Markov Decision Processes (MDPs): UCB-Advantage (Zhang et al. 2020) and Q-EarlySettled-Advantage (Li et al. 2021). UCB-Advantage and Q-EarlySettled-Advantage improve upon the results based on Hoeffding-type bonuses and achieve the almost optimal $\sqrt{T}$-type regret bound in the worst-case scenario, where $T$ is the total number of steps. However, the benign structures of the MDPs such as a strictly positive suboptimality gap can significantly improve the regret. While gap-dependent regret bounds have been obtained for Q-learning with Hoeffding-type bonuses, it remains an open question to establish gap-dependent regret bounds for Q-learning using variance estimators in their bonuses and reference-advantage decomposition for variance reduction. We develop a novel error decomposition framework to prove gap-dependent regret bounds of UCB-Advantage and Q-EarlySettled-Advantage that are logarithmic in $T$ and improve upon existing ones for Q-learning algorithms. Moreover, we establish the gap-dependent bound for the policy switching cost of UCB-Advantage and improve that under the worst-case MDPs. To our knowledge, this paper presents the first gap-dependent regret analysis for Q-learning using variance estimators and reference-advantage decomposition and also provides the first gap-dependent analysis on policy switching cost for Q-learning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。