QPLEX：双重对偶多智能体Q学习
QPLEX: Duplex Dueling Multi-Agent Q-Learning
摘要：
我们探索基于值的多智能体强化学习（MARL）在流行的集中训练与分散执行（CTDE）的范式。CTDE有一个重要的概念，个体-全局-最大（IGM）原则，它要求联合和局部行动选择之间的一致性，以支持有效的局部决策。然而，为了实现可扩展性，现有的MARL方法要么限制其价值函数类的表示表达性，要么放松IGM一致性，这可能遭受不稳定风险，或者在复杂域中可能不能很好地执行。本文提出了一种新的MARL方法，称为duPLEX决斗多智能体Q学习（QPLEX），它采用双工决斗网络架构来分解联合价值函数。这种双重决斗结构将IGM原理编码到神经网络架构中，从而实现有效的价值函数学习。理论分析表明，QPLEX实现了完整的IGM函数类。对星际争霸II微观管理任务的实证实验表明，QPLEX在在线和离线数据汇集设置中的表现明显优于最先进的基线，并且还显示QPLEX实现了高样本效率，并且可以从离线数据集中受益，而无需额外的在线探索。
英文摘要：
We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efficient local decision-making. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or may not perform well in complex domains. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efficient value function learning. Theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline data collection settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional online exploration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
