arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.00356cs.AI
收藏

QFree:一种面向多智能体强化学习的通用值函数分解方法

QFree: A Universal Value Function Factorization for Multi-Agent Reinforcement Learning

Northwestern Polytechnical University(西北工业大学)

Rizhong Wang, Huiping Li, Di Cui, Demin Xu

AI总结:

本文提出QFree通用值函数分解方法,通过优势函数建立IGM原则的数学等价条件并设计更具表达能力的混合网络,在非单调矩阵博弈和SMAC基准中取得领先性能。

摘要:

集中式训练在多智能体强化学习(MARL)领域中被广泛用于保证训练过程的稳定性。一旦获得联合策略,设计一种值函数分解方法以提取智能体的最优分散式策略就至关重要,该方法需要满足个体-全局最大值(IGM)原则。虽然对IGM函数类施加额外限制有助于满足要求,但这会以限制其在更复杂的多智能体环境中的应用为代价。在本文中,我们提出了QFree,一种面向MARL的通用值函数分解方法。我们首先基于优势函数建立IGM原则的数学等价条件,这确保了该原则在没有任何妥协的情况下成立,消除了传统方法的保守性。然后,我们建立了一种更具表达能力的混合网络架构,能够实现等价分解。特别地,通过在MARL算法的策略评估过程中将等价条件作为正则化项,开发了新的损失函数。最后,在非单调矩阵博弈场景中验证了所提方法的有效性。此外,我们表明QFree在通用复杂MARL基准环境——星际争霸多智能体挑战(SMAC)中达到了最先进的性能。

英文摘要:

Centralized training is widely utilized in the field of multi-agent reinforcement learning (MARL) to assure the stability of training process. Once a joint policy is obtained, it is critical to design a value function factorization method to extract optimal decentralized policies for the agents, which needs to satisfy the individual-global-max (IGM) principle. While imposing additional limitations on the IGM function class can help to meet the requirement, it comes at the cost of restricting its application to more complex multi-agent environments. In this paper, we propose QFree, a universal value function factorization method for MARL. We start by developing mathematical equivalent conditions of the IGM principle based on the advantage function, which ensures that the principle holds without any compromise, removing the conservatism of conventional methods. We then establish a more expressive mixing network architecture that can fulfill the equivalent factorization. In particular, the novel loss function is developed by considering the equivalent conditions as regularization term during policy evaluation in the MARL algorithm. Finally, the effectiveness of the proposed method is verified in a nonmonotonic matrix game scenario. Moreover, we show that QFree achieves the state-of-the-art performance in a general-purpose complex MARL benchmark environment, Starcraft Multi-Agent Challenge (SMAC).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑