Centralized training is widely utilized in the field of multi-agent reinforcement learning (MARL) to assure the stability of training process. Once a joint policy is obtained, it is critical to design a value function factorization method to extract optimal decentralized policies for the agents, which needs to satisfy the individual-global-max (IGM) principle. While imposing additional limitations on the IGM function class can help to meet the requirement, it comes at the cost of restricting its application to more complex multi-agent environments. In this paper, we propose QFree, a universal value function factorization method for MARL. We start by developing mathematical equivalent conditions of the IGM principle based on the advantage function, which ensures that the principle holds without any compromise, removing the conservatism of conventional methods. We then establish a more expressive mixing network architecture that can fulfill the equivalent factorization. In particular, the novel loss function is developed by considering the equivalent conditions as regularization term during policy evaluation in the MARL algorithm. Finally, the effectiveness of the proposed method is verified in a nonmonotonic matrix game scenario. Moreover, we show that QFree achieves the state-of-the-art performance in a general-purpose complex MARL benchmark environment, Starcraft Multi-Agent Challenge (SMAC).
Breakthrough the Suboptimal Stable Point in Value-Factorization-Based Multi-Agent Reinforcement Learning
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University(西安交通大学人工智能与机器人研究所,人机混合增强智能国家重点实验室);Zhongguancun Academy(中关村学院)
Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);Stanford University(斯坦福大学);MOE Key Lab of Embedded System and Service Computing, Tongji University, Shanghai, China(同济大学嵌入式系统与服务计算教育部重点实验室,上海,中国);The University of Hong Kong(香港大学);Shanghai Innovation Institute(上海创新研究院)
Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative Multi-Agent Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
针对合作多智能体强化学习中IGM原则限制问题,提出双自感知价值分解框架,结合自我策略与 alter ego 价值函数并引入反自我探索机制,实现完全无需IGM的价值分解并在多任务中表现优异。
School of Computer and Information Technology, Beijing Jiaotong University(北京交通大学计算机与信息技术学院);Beijing Key Laboratory of Traffic Data Analysis and Mining, Beijing Jiaotong University(北京交通大学交通数据分析与挖掘北京市重点实验室)
RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learning
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University(清华大学交叉信息研究院);Department of Computer Science, University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机科学系);Harvard University(哈佛大学)
Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus
The Hong Kong University of Science and Technology(香港科学与技术大学);The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))