arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.08844cs.GTmath.OC
收藏

多智能体强化学习中的均衡选择:一个统一的框架

Equilibrium Selection for Multi-agent Reinforcement Learning: A Unified Framework

Massachusetts Institute of Technology(麻省理工学院);UIUC, Department of Industrial and Enterprise Systems Engineering(伊利诺伊大学厄巴纳-香槟分校工业与企业系统工程系);Harvard University, School of Engineering and Applied Sciences(哈佛大学工程与应用科学学院)

Runyu Zhang, Gioele Zardini, Asuman Ozdaglar, Jeff Shamma, Na Li

AI总结:

本文提出了一种统一的actor-critic框架,用于在多智能体强化学习中选择具有高社会福利的均衡。

摘要:

尽管多智能体强化学习(MARL)已经产生了许多收敛到纳什或相关均衡的算法,但这些均衡往往不唯一,并且可以表现出显著不同的效率。这提出了一个根本性的问题:如何设计学习动态,使其不仅收敛到均衡,还能选择具有良好性能的均衡,例如高社会福利?与MARL文献不同,均衡选择在正常形式游戏中已被广泛研究,其中去中心化动态已知收敛于潜在最大化或帕累托最优纳什均衡(NEs)。受这些结果的启发,我们研究了有限时间随机游戏中的均衡选择。我们提出了一种统一的actor-critic框架,在其中批评者学习状态-动作价值函数,而actor应用一个经典的均衡选择规则逐状态进行,将学习到的价值视为阶段游戏支付。我们证明,在标准的随机稳定性假设下,所得到动态的随机稳定策略继承了底层正常形式学习规则的均衡选择属性。作为结果,我们获得了马尔可夫潜在游戏中的潜在最大化策略,以及一般和游戏中的帕累托最优(马尔可夫完美)均衡,以及框架的样本实现。

英文摘要:

While multi-agent reinforcement learning (MARL) has produced numerous algorithms that converge to Nash or related equilibria, such equilibria are often non-unique and can exhibit widely varying efficiency. This raises a fundamental question: how can one design learning dynamics that not only converge to equilibrium but also select equilibria with desirable performance, such as high social welfare? In contrast to the MARL literature, equilibrium selection has been extensively studied in normal-form games, where decentralized dynamics are known to converge to potential-maximizing or Pareto-optimal Nash equilibria (NEs). Motivated by these results, we study equilibrium selection in finite-horizon stochastic games. We propose a unified actor-critic framework in which a critic learns state-action value functions, and an actor applies a classical equilibrium-selection rule state-wise, treating learned values as stage-game payoffs. We show that, under standard stochastic stability assumptions, the stochastically stable policies of the resulting dynamics inherit the equilibrium selection properties of the underlying normal-form learning rule. As consequences, we obtain potential-maximizing policies in Markov potential games and Pareto-optimal (Markov perfect) equilibria in general-sum stochastic games, together with sample-based implementation of the framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑