arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2006.12007cs.LGcs.AIstat.ML
收藏

具有自学习的近优强化学习

Near-Optimal Reinforcement Learning with Self-Play

Salesforce Research(Salesforce研究院);Princeton University(普林斯顿大学);MIT(麻省理工学院)

Yu Bai, Chi Jin, Tiancheng Yu

摘要:

研究了二人零和博弈中强化学习的最优算法设计问题。我们专注于自我发挥算法,学习最优策略,通过发挥自己没有任何直接的监督。在具有$S$状态、$A$最大参与者行动和$B$最小参与者行动的表格式情景马尔可夫博弈中,当仅突出对$(S,A,B)$的依赖性时,找到近似纳什均衡的最佳现有算法需要$\tilde{\mathcal{O}}(S^2AB)$博弈步骤。相比之下,现有的最佳下限尺度为$Ω(S(A+B))$,并且与上限有显著的差距。本文关闭这一差距的第一次:我们提出了一个乐观的\emph{Nash Q-learning}算法的样本复杂度$\tilde{\mathcal{O}}(SAB)$的变体,和一个新的\emph{Nash V-learning}算法的样本复杂度$\tilde{\mathcal{O}}(S(A+B))$。后一个结果匹配的信息理论的下限,在所有的问题相关的参数,除了一个多项式因子的长度,每个情节。此外,我们提出了一个计算硬度的结果,学习最佳对策对一个固定的对手在马尔可夫游戏-学习目标不同,从寻找纳什均衡。

英文摘要:

This paper considers the problem of designing optimal algorithms for reinforcement learning in two-player zero-sum games. We focus on self-play algorithms which learn the optimal policy by playing against itself without any direct supervision. In a tabular episodic Markov game with $S$ states, $A$ max-player actions and $B$ min-player actions, the best existing algorithm for finding an approximate Nash equilibrium requires $\tilde{\mathcal{O}}(S^2AB)$ steps of game playing, when only highlighting the dependency on $(S,A,B)$. In contrast, the best existing lower bound scales as $Ω(S(A+B))$ and has a significant gap from the upper bound. This paper closes this gap for the first time: we propose an optimistic variant of the \emph{Nash Q-learning} algorithm with sample complexity $\tilde{\mathcal{O}}(SAB)$, and a new \emph{Nash V-learning} algorithm with sample complexity $\tilde{\mathcal{O}}(S(A+B))$. The latter result matches the information-theoretic lower bound in all problem-dependent parameters except for a polynomial factor of the length of each episode. In addition, we present a computational hardness result for learning the best responses against a fixed opponent in Markov games---a learning objective different from finding the Nash equilibrium.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑