arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.02635cs.AI
收藏

PPS-QMIX:用于加速多智能体强化学习收敛的周期性参数共享方法

PPS-QMIX: Periodically Parameter Sharing for Accelerating Convergence of Multi-Agent Reinforcement Learning

China University of Petroleum(中国石油大学);Télécom Paris(巴黎电信学院)

Ke Zhang, DanDan Zhu, Qiuhan Xu, Hao Zhou, Ce Zheng

AI总结:

针对多智能体强化学习训练慢的问题,受联邦学习启发提出三种周期性参数共享机制,应用于QMIX后在SMAC环境上实现10%-30%的平均性能提升,还能完成原方法无法取胜的任务。

摘要:

多智能体强化学习(MARL)的训练是一个耗时的过程,其根源在于每个智能体的分布偏移。其中一个缺陷是,MARL中每个智能体的策略是独立的,但实际上它们处于协作关系中。因此,多智能体强化学习中的一个核心问题是如何高效加速训练过程。为解决这一问题,现有研究利用跨多智能体的中心化函数(CF)来学习团队奖励对每个智能体的贡献。然而,基于CF的方法在价值网络的估计中会引入来自其他智能体的联合误差。受联邦学习启发,我们提出了三种简单新颖的机制,分别称为平均周期性参数共享(A-PPS)、奖励缩放周期性参数共享(RS-PPS)和部分个性化周期性参数共享(PP-PPS),用于加速MARL的训练。智能体在训练过程中周期性共享Q值网络;具有相同身份的智能体将收集到的奖励作为缩放因子,并在周期内更新部分神经网络以共享不同参数。我们将所提方法应用于经典MARL方法QMIX,并在星际争霸多智能体挑战(SMAC)环境的各类任务上进行评估。数值实验结果显示性能得到大幅提升,平均改进幅度达10%-30%,并且能够完成QMIX无法取胜的任务。我们的代码可从https://github.com/ColaZhang22/PPS-QMIX下载。

英文摘要:

Training for multi-agent reinforcement learning(MARL) is a time-consuming process caused by distribution shift of each agent. One drawback is that strategy of each agent in MARL is independent but actually in cooperation. Thus, a vertical issue in multi-agent reinforcement learning is how to efficiently accelerate training process. To address this problem, current research has leveraged a centralized function(CF) across multiple agents to learn contribution of the team reward for each agent. However, CF based methods introduce joint error from other agents in estimation of value network. In so doing, inspired by federated learning, we propose three simple novel approaches called Average Periodically Parameter Sharing(A-PPS), Reward-Scalability Periodically Parameter Sharing(RS-PPS) and Partial Personalized Periodically Parameter Sharing(PP-PPS) mechanism to accelerate training of MARL. Agents share Q-value network periodically during the training process. Agents which has same identity adapt collected reward as scalability and update partial neural network during period to share different parameters. We apply our approaches in classical MARL method QMIX and evaluate our approaches on various tasks in StarCraft Multi-Agent Challenge(SMAC) environment. Performance of numerical experiments yield enormous enhancement, with an average improvement of 10\%-30\%, and enable to win tasks that QMIX cannot. Our code can be downloaded from https://github.com/ColaZhang22/PPS-QMIX

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑