FM3Q: 面向两队零和马尔可夫博弈的因式分解多智能体MiniMax Q学习
FM3Q: Factorized Multi-Agent MiniMax Q-Learning for Two-Team Zero-Sum Markov Game
AI总结：
针对两队零和马尔可夫博弈现有方法效率低的问题，提出基于IGMM准则的FM3Q强化学习框架，分解联合Q函数并迭代求解，配套在线算法获取去中心化策略，理论证明收敛且实验验证性能更优。
摘要：
许多现实应用中的智能体可分为两个队伍，同队智能体的收益相同，而对阵双方的收益符号相反，这类问题被称为两队零和马尔可夫博弈（2t0sMGs），近年来可通过强化学习求解。但现有方法因未充分考虑队內信用分配、数据利用效率且存在计算难解性问题，整体效率较低。本文提出个体-全局-极小极大（IGMM）准则，通过2t0sMGs中的Q函数保证两队极小极大行为与个体贪婪行为的一致性。基于该准则，我们提出一种新型多智能体强化学习框架——因式分解多智能体MiniMax Q学习（FM3Q），它可将联合极小极大Q函数分解为个体Q函数，并迭代求解满足IGMM准则的2t0sMGs极小极大Q函数。此外，本文提出一种基于神经网络的在线学习算法实现FM3Q，为两队玩家获取确定性的去中心化极小极大策略。理论分析证明了FM3Q的收敛性。实验中我们在三个环境中评估了FM3Q的学习效率与最终性能，验证了其在2t0sMGs问题上的优越性。
英文摘要：
Many real-world applications involve some agents that fall into two teams, with payoffs that are equal within the same team but of opposite sign across the opponent team. The so-called two-team zero-sum Markov games (2t0sMGs) can be resolved with reinforcement learning in recent years. However, existing methods are thus inefficient in light of insufficient consideration of intra-team credit assignment, data utilization and computational intractability. In this paper, we propose the individual-global-minimax (IGMM) principle to ensure the coherence between two-team minimax behaviors and the individual greedy behaviors through Q functions in 2t0sMGs. Based on it, we present a novel multi-agent reinforcement learning framework, Factorized Multi-Agent MiniMax Q-Learning (FM3Q), which can factorize the joint minimax Q function into individual ones and iteratively solve for the IGMM-satisfied minimax Q functions for 2t0sMGs. Moreover, an online learning algorithm with neural networks is proposed to implement FM3Q and obtain the deterministic and decentralized minimax policies for two-team players. A theoretical analysis is provided to prove the convergence of FM3Q. Empirically, we use three environments to evaluate the learning efficiency and final performance of FM3Q and show its superiority on 2t0sMGs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
