arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.08612cs.LGcs.AIcs.MA
收藏

解决多Agent策略梯度的差异

Settling the Variance of Multi-Agent Policy Gradients

Imperial College London(帝国理工学院);Huawei R&D UK(华为英国研发中心);Shanghai Jiao Tong University(上海交通大学);Institute of Automation, Chinese Academy of Science(中国科学院自动化研究所);University College London(伦敦大学学院);Institute for AI, Peking University(北京大学人工智能研究院)

Jakub Grudzien Kuba, Muning Wen, Yaodong Yang, Linghui Meng, Shangding Gu, Haifeng Zhang, David Henry Mguni, Jun Wang

摘要:

策略梯度(PG)方法是流行的强化学习(RL)方法,其中通常应用基线来减小梯度估计的方差。在多智能体强化学习(MARL)中,虽然PG定理可以自然地扩展,但多智能体PG(MAPG)方法的有效性会随着梯度估计的方差随着智能体数量的增加而迅速下降。在本文中,我们提供了一个严格的分析MAPG方法,首先,量化的贡献的代理和代理的探索MAPG估计方差的数量。基于此分析,我们推导出最佳基线(OB),实现最小方差。在比较OB,我们衡量现有的MARL算法,如香草MAPG和COMA的过剩方差。考虑到使用深度神经网络,我们还提出了OB的代理版本,它可以无缝地插入到MARL中任何现有的PG方法中。在多智能体MuJoCo和星际争霸挑战的基准测试中,我们的OB技术有效地稳定了训练,并显著提高了多智能体PPO和COMA算法的性能。

英文摘要:

Policy gradient (PG) methods are popular reinforcement learning (RL) methods where a baseline is often applied to reduce the variance of gradient estimates. In multi-agent RL (MARL), although the PG theorem can be naturally extended, the effectiveness of multi-agent PG (MAPG) methods degrades as the variance of gradient estimates increases rapidly with the number of agents. In this paper, we offer a rigorous analysis of MAPG methods by, firstly, quantifying the contributions of the number of agents and agents' explorations to the variance of MAPG estimators. Based on this analysis, we derive the optimal baseline (OB) that achieves the minimal variance. In comparison to the OB, we measure the excess variance of existing MARL algorithms such as vanilla MAPG and COMA. Considering using deep neural networks, we also propose a surrogate version of OB, which can be seamlessly plugged into any existing PG methods in MARL. On benchmarks of Multi-Agent MuJoCo and StarCraft challenges, our OB technique effectively stabilises training and improves the performance of multi-agent PPO and COMA algorithms by a significant margin.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑