arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.01939cs.AIcs.MA
收藏

NQMIX:深度多智能体强化学习的非单调价值函数分解

NQMIX: Non-monotonic Value Function Factorization for Deep Multi-Agent Reinforcement Learning

Quanlin Chen

摘要:

近年来,基于多主体价值的方法取得了很大的进展,尤其是价值分解方法。然而,价值函数分解仍存在许多局限性。在VDN中,联合作用-价值函数是各主体作用-价值函数的和,而QMIX的联合作用-价值函数是各主体作用-价值函数的单调混合。QTRAN在一定程度上减少了联合作用--可表示的价值函数的限制,但在复杂任务中的性能并不令人满意。在本文中,为了扩展类的联合价值函数,可以表示,我们提出了一种新的演员批评的方法称为NQMIX。NQMIX在QMIX的基础上引入了一个偏离策略的策略梯度,并修改了QMIX的网络结构,消除了QMIX的单调性约束,实现了联合动作-价值函数的非单调价值函数分解。此外,NQMIX以状态值作为学习目标,克服了QMIX中学习目标被高估的问题。此外,NQMIX可以通过在自身上引入确定性策略梯度来扩展到连续动作空间设置。最后,我们在SMAC域上评估了我们的actor-critic方法,并表明它在具有异构代理类型的复杂地图上具有比COMA和QMIX更强的性能。此外,我们的烧蚀结果表明我们对混合器的改进是有效的。

英文摘要:

Multi-agent value-based approaches recently make great progress, especially value decomposition methods. However, there are still a lot of limitations in value function factorization. In VDN, the joint action-value function is the sum of per-agent action-value function while the joint action-value function of QMIX is the monotonic mixing of per-agent action-value function. To some extent, QTRAN reduces the limitation of joint action-value functions that can be represented, but it has unsatisfied performance in complex tasks. In this paper, in order to extend the class of joint value functions that can be represented, we propose a novel actor-critic method called NQMIX. NQMIX introduces an off-policy policy gradient on QMIX and modify its network architecture, which can remove the monotonicity constraint of QMIX and implement a non-monotonic value function factorization for the joint action-value function. In addition, NQMIX takes the state-value as the learning target, which overcomes the problem in QMIX that the learning target is overestimated. Furthermore, NQMIX can be extended to continuous action space settings by introducing deterministic policy gradient on itself. Finally, we evaluate our actor-critic methods on SMAC domain, and show that it has a stronger performance than COMA and QMIX on complex maps with heterogeneous agent types. In addition, our ablation results show that our modification of mixer is effective.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑