arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.00080cs.LGstat.ML
收藏

在单侧反馈/全反馈设置中可证明更有效的Q学习

Provably More Efficient Q-Learning in the One-Sided-Feedback/Full-Feedback Settings

Massachusetts Institute of Technology(麻省理工学院)

Xiao-Yue Gong, David Simchi-Levi

摘要:

出于情节版本的经典库存控制问题,我们提出了一个新的Q学习算法,消除基于半Q学习(HQL),享有提高效率的各种各样的问题,在单边反馈设置现有的算法。我们还提供了一个更简单的变体的算法,全Q学习(FQL),全反馈设置。我们建立了HQL招致$ \tilde{\mathcal{O}}(H^3\sqrt{ T})$后悔和FQL招致$\tilde{\mathcal{O}}(H^2\sqrt{ T})$后悔,其中$H$是每个情节的长度,$T$是视界的总长度。遗憾边界不受可能的巨大状态和行动空间的影响。我们的数值实验证明了HQL和FQL的优越效率,以及将强化学习与更丰富的反馈模型相结合的潜力。

英文摘要:

Motivated by the episodic version of the classical inventory control problem, we propose a new Q-learning-based algorithm, Elimination-Based Half-Q-Learning (HQL), that enjoys improved efficiency over existing algorithms for a wide variety of problems in the one-sided-feedback setting. We also provide a simpler variant of the algorithm, Full-Q-Learning (FQL), for the full-feedback setting. We establish that HQL incurs $ \tilde{\mathcal{O}}(H^3\sqrt{ T})$ regret and FQL incurs $\tilde{\mathcal{O}}(H^2\sqrt{ T})$ regret, where $H$ is the length of each episode and $T$ is the total length of the horizon. The regret bounds are not affected by the possibly huge state and action space. Our numerical experiments demonstrate the superior efficiency of HQL and FQL, and the potential to combine reinforcement learning with richer feedback models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑