arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.14280cs.GTcs.AIcs.LG
收藏

多人随机博弈中的学习

Learning in Multi-Player Stochastic Games

Columbia University(哥伦比亚大学)

William Brown

摘要:

本文研究了有限时间内多局中人随机博弈中的同时学习问题。虽然随机博弈的典型目标解是一个纳什均衡,但这对许多参与者来说是难以解决的。相反,我们专注于{ \it相关均衡}的变体,如那些广泛的形式的游戏研究。我们从对抗性MDP问题的一个困难结果开始:即使对于3的视野,获得对最佳非平稳策略的次线性遗憾也是\textsf{NP} -当奖励和过渡都是对抗性的时。这意味着,收敛到最弱的自然解概念-正规形式的粗相关equilibrium-是不可能的,通过黑盒减少到一个没有遗憾的算法,即使在随机游戏与恒定的地平线(除非$\textsf{NP}\subseteq\textsf{BPP}$)。相反,我们转向一个不同的目标:当所有参与者使用这些算法时,{ \it生成}一个均衡。我们的主要结果是算法产生的{ \it extensive-form}相关的平衡,其运行时间是指数的地平线,但多项式的所有其他参数。我们给出了一个类似的算法,这是多项式的所有参数的“快速混合”随机游戏。我们还展示了一种方法,有效地达到正常形式的粗相关平衡的“单控制器”随机游戏,遵循传统的无悔的方法。当共享的随机性是可用的,这两个生成算法可以扩展到同时遗憾界和收敛在传统意义上。

英文摘要:

We consider the problem of simultaneous learning in stochastic games with many players in the finite-horizon setting. While the typical target solution for a stochastic game is a Nash equilibrium, this is intractable with many players. We instead focus on variants of {\it correlated equilibria}, such as those studied for extensive-form games. We begin with a hardness result for the adversarial MDP problem: even for a horizon of 3, obtaining sublinear regret against the best non-stationary policy is \textsf{NP}-hard when both rewards and transitions are adversarial. This implies that convergence to even the weakest natural solution concept -- normal-form coarse correlated equilbrium -- is not possible via black-box reduction to a no-regret algorithm even in stochastic games with constant horizon (unless $\textsf{NP}\subseteq\textsf{BPP}$). Instead, we turn to a different target: algorithms which {\it generate} an equilibrium when they are used by all players. Our main result is algorithm which generates an {\it extensive-form} correlated equilibrium, whose runtime is exponential in the horizon but polynomial in all other parameters. We give a similar algorithm which is polynomial in all parameters for "fast-mixing" stochastic games. We also show a method for efficiently reaching normal-form coarse correlated equilibria in "single-controller" stochastic games which follows the traditional no-regret approach. When shared randomness is available, the two generative algorithms can be extended to give simultaneous regret bounds and converge in the traditional sense.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑