Multiagent learning is a necessary yet challenging problem as multiagent systems become more prevalent and environments become more dynamic. Much of the groundbreaking work in this area draws on notable results from game theory, in particular, the concept of Nash equilibria. Learners that directly learn an equilibrium obviously rely on their existence. Learners that instead seek to play optimally with respect to the other players also depend upon equilibria since equilibria are fixed points for learning. From another perspective, agents with limitations are real and common. These may be undesired physical limitations as well as self-imposed rational limitations, such as abstraction and approximation techniques, used to make learning tractable. This article explores the interactions of these two important concepts: equilibria and limitations in learning. We introduce the question of whether equilibria continue to exist when agents have limitations. We look at the general effects limitations can have on agent behavior, and define a natural extension of equilibria that accounts for these limitations. Using this formalization, we make three major contributions: (i) a counterexample for the general existence of equilibria with limitations, (ii) sufficient conditions on limitations that preserve their existence, (iii) three general classes of games and limitations that satisfy these conditions. We then present empirical results from a specific multiagent learning algorithm applied to a specific instance of limited agents. These results demonstrate that learning with limitations is feasible, when the conditions outlined by our theoretical analysis hold.
Convex Markov Games and Beyond: New Proof of Existence, Characterization and Learning Algorithms for Nash Equilibria
Singapore University of Technology and Design（新加坡科技设计大学）；University of California Irvine, USA（加州大学 Irvine 分校）；Centre for Quantum Technologies, National University of Singapore（新加坡国立大学量子中心）；Archimedes/Athena Research Center, Greece（阿基米德/雅典娜研究中心）
On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learning
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；DeepMind（深度思维）；Massachusetts Institute of Technology（麻省理工学院）
多智能体强化学习（MARL）算法经常遭受指数样本复杂度对智能体数量的依赖，这种现象被称为\emph{the curse of multiagents}。在本文中，我们解决这一挑战，调查样本效率的无模型算法在\emph{decentralized} MARL，并旨在改善现有的算法沿这条线。对于学习（粗）一般和马尔可夫博弈中的相关均衡，我们提出了\emph{stage-based} V-学习算法，大大简化了最近的作品的算法设计和分析，并规避了一个相当复杂的无\emph{weighted}遗憾强盗子程序。针对马尔可夫势博弈中纳什均衡的学习问题，提出了一种独立的策略梯度算法，该算法采用分散的基于动量的方差缩减技术。我们所有的算法都是去中心化的，因为每个代理只能根据其本地信息做出决策。在学习过程中既不需要通信，也不需要集中协调，从而导致对大量代理的自然泛化。我们还提供了数值模拟来证实我们的理论研究结果。
