arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.03094cs.LGcs.AIcs.GTcs.MAstat.ML
收藏

结合无悔和Q学习

Combining No-regret and Q-learning

University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Microsoft Research(微软研究院)

Ian A. Kash, Michael Sullins, Katja Hofmann

摘要:

反事实后悔最小化(CFR)在扑克等既有终端状态又有完美回忆的环境中取得了成功。我们试图了解如何放宽这些要求。作为第一步,我们介绍了一个简单的算法,局部无遗憾学习(LONR),它使用一个Q学习的更新规则,允许学习没有终端状态或完美的回忆。我们证明了它的收敛的基本情况下的MDPs(和有限的扩展),目前的实证结果表明,它实现了最后的收敛在一些设置,最值得注意的是诺维茨基游戏,一类马尔可夫游戏专门设计的学习是具有挑战性的,没有事先的算法是已知的,甚至平均达到收敛到一个固定的平衡。

英文摘要:

Counterfactual Regret Minimization (CFR) has found success in settings like poker which have both terminal states and perfect recall. We seek to understand how to relax these requirements. As a first step, we introduce a simple algorithm, local no-regret learning (LONR), which uses a Q-learning-like update rule to allow learning without terminal states or perfect recall. We prove its convergence for the basic case of MDPs (and limited extensions of them) and present empirical results showing that it achieves last iterate convergence in a number of settings, most notably NoSDE games, a class of Markov games specifically designed to be challenging to learn where no prior algorithm is known to achieve convergence to a stationary equilibrium even on average.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑