arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.12181cs.GTcs.LGcs.MA
收藏

凸马尔可夫游戏及更进一步:Nash均衡的存在性、特征及学习算法的新证明

Convex Markov Games and Beyond: New Proof of Existence, Characterization and Learning Algorithms for Nash Equilibria

Singapore University of Technology and Design(新加坡科技设计大学);University of California Irvine, USA(加州大学 Irvine 分校);Centre for Quantum Technologies, National University of Singapore(新加坡国立大学量子中心);Archimedes/Athena Research Center, Greece(阿基米德/雅典娜研究中心)

Anas Barakat, Ioannis Panageas, Antonios Varvitsiotis

AI总结:

本文提出通用效用马尔可夫游戏(GUMGs)理论框架,证明Nash均衡的存在性与一阶静态点一致,并设计无模型策略梯度算法,扩展了传统cMGs的理论边界。

摘要:

凸马尔可夫游戏(cMGs)最近被引入为一种广泛的多智能体学习问题,它将马尔可夫游戏扩展到战略代理优化一般效用而非加性奖励的设置。尽管cMGs扩展了建模前沿,但其理论基础,特别是Nash均衡(NE)的结构和学习算法的保证尚未充分理解。在本文中,我们针对一种扩展的cMGs,称为通用效用马尔可夫游戏(GUMGs),解决了这些差距。我们证明在GUMGs中,Nash均衡与投影伪梯度动力学的不动点(即一阶静态点)一致,这得益于一种新的代理梯度支配性质。这一见解还提供了使用布劳威尔不动点定理证明NE存在的简单证明。我们进一步展示了马尔可夫完美均衡的存在性。基于这一特征,我们为GUMGs建立了策略梯度定理并设计了无模型策略梯度算法。对于潜在GUMGs,我们建立了在精确梯度下计算近似NE的迭代复杂度保证,并在生成模型和on-policy设置中提供了样本复杂度界限。我们的结果超越了先前工作仅限于零和cMGs的范围,为共同利益cMGs提供了首次理论分析。

英文摘要:

Convex Markov Games (cMGs) were recently introduced as a broad class of multi-agent learning problems that generalize Markov games to settings where strategic agents optimize general utilities beyond additive rewards. While cMGs expand the modeling frontier, their theoretical foundations, particularly the structure of Nash equilibria (NE) and guarantees for learning algorithms, are not yet well understood. In this work, we address these gaps for an extension of cMGs, which we term General Utility Markov Games (GUMGs), capturing new applications requiring coupling between agents' occupancy measures. We prove that in GUMGs, Nash equilibria coincide with the fixed points of projected pseudo-gradient dynamics (i.e., first-order stationary points), enabled by a novel agent-wise gradient domination property. This insight also yields a simple proof of NE existence using Brouwer's fixed-point theorem. We further show the existence of Markov perfect equilibria. Building on this characterization, we establish a policy gradient theorem for GUMGs and design a model-free policy gradient algorithm. For potential GUMGs, we establish iteration complexity guarantees for computing approximate-NE under exact gradients and provide sample complexity bounds in both the generative model and on-policy settings. Our results extend beyond prior work restricted to zero-sum cMGs, providing the first theoretical analysis of common-interest cMGs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑