arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09946cs.LGcs.SYeess.SY
收藏

有限时间分析的同时双Q学习

Finite-Time Analysis of Simultaneous Double Q-learning

Department of Electrical Engineering, Korea Advanced Institute of Science(韩国科学技术院电子工程系)

Hyunjun Na, Donghwan Lee

AI总结:

本文提出了一种改进的双Q学习算法SDQ,通过切换系统框架实现有限时间分析,提升收敛速度并保持缓解过估计偏差的能力。

摘要:

Q学习是强化学习(RL)中最基本的算法之一。尽管其在各种应用中广泛成功,但其在Q学习更新过程中容易产生过估计偏差。为了解决这个问题,双Q学习使用两个独立的Q估计器,在学习过程中随机选择和更新。本文提出了一种改进的双Q学习,称为同时双Q学习(SDQ),并对其进行了有限时间分析。SDQ消除了在两个Q估计器之间随机选择的需要,这种修改使我们能够通过新颖的切换系统框架对双Q学习进行有限时间分析。实证研究显示,SDQ比双Q学习收敛得更快,同时保留了缓解最大化偏差的能力。最后,我们为SDQ推导出有限时间的期望误差界。

英文摘要:

$Q$-learning is one of the most fundamental reinforcement learning (RL) algorithms. Despite its widespread success in various applications, it is prone to overestimation bias in the $Q$-learning update. To address this issue, double $Q$-learning employs two independent $Q$-estimators which are randomly selected and updated during the learning process. This paper proposes a modified double $Q$-learning, called simultaneous double $Q$-learning (SDQ), with its finite-time analysis. SDQ eliminates the need for random selection between the two $Q$-estimators, and this modification allows us to analyze double $Q$-learning through the lens of a novel switching system framework facilitating efficient finite-time analysis. Empirical studies demonstrate that SDQ converges faster than double $Q$-learning while retaining the ability to mitigate the maximization bias. Finally, we derive a finite-time expected error bound for SDQ.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑