arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.00285cs.AIcs.MA
收藏

基于Shapley反事实积分的多智能体强化学习

Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning

Zhejiang University(浙江大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳市人工智能与机器人研究院)

Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, Jun Xiao

摘要:

集中训练与分散执行(CTDE)是协作多智能体强化学习(MARL)环境中的一种流行范式,并在许多实际应用中得到广泛应用。训练过程中的一个主要挑战是信用分配,其目的是根据全局奖励来推断每个代理的贡献。现有的信用分配方法集中于将联合价值函数分解为单个价值函数或测量局部观测和动作对全局价值函数的影响。这些方法缺乏对多个代理之间复杂交互的全面考虑,导致不适当的信用分配,从而导致MARL结果平庸。我们提出了Shapley反事实信用分配,一种新的方法,明确的信用分配占联盟的代理。具体而言,Shapley值及其所需属性在深度MARL中被利用来对任何代理组合进行信贷,这使我们能够估计每个代理的个人信贷。尽管有这种能力,主要的技术困难在于Shapley值的计算复杂性,Shapley值随着代理的数量而因子地增长。相反,我们利用近似方法通过蒙特卡罗采样,这降低了样本的复杂性,同时保持其有效性。我们评估我们的方法在星际争霸II基准在不同的情况下。我们的方法优于现有的合作MARL算法显着,达到了最先进的,特别是大利润率的任务,更严重的困难。

英文摘要:

Centralized Training with Decentralized Execution (CTDE) has been a popular paradigm in cooperative Multi-Agent Reinforcement Learning (MARL) settings and is widely used in many real applications. One of the major challenges in the training process is credit assignment, which aims to deduce the contributions of each agent according to the global rewards. Existing credit assignment methods focus on either decomposing the joint value function into individual value functions or measuring the impact of local observations and actions on the global value function. These approaches lack a thorough consideration of the complicated interactions among multiple agents, leading to an unsuitable assignment of credit and subsequently mediocre results on MARL. We propose Shapley Counterfactual Credit Assignment, a novel method for explicit credit assignment which accounts for the coalition of agents. Specifically, Shapley Value and its desired properties are leveraged in deep MARL to credit any combinations of agents, which grants us the capability to estimate the individual credit for each agent. Despite this capability, the main technical difficulty lies in the computational complexity of Shapley Value who grows factorially as the number of agents. We instead utilize an approximation method via Monte Carlo sampling, which reduces the sample complexity while maintaining its effectiveness. We evaluate our method on StarCraft II benchmarks across different scenarios. Our method outperforms existing cooperative MARL algorithms significantly and achieves the state-of-the-art, with especially large margins on tasks with more severe difficulties.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑