重思协作式多智能体强化学习中的实现技巧与单调性约束
Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
AI总结：
本文考察QMIX变体的代码级优化与单调性约束，发现归一化优化的QMIX性能更优，且单调性约束可提高SMAC和DEPP的样本效率并给出理论解释。
摘要：
许多复杂的多智能体系统，例如机器人群控制和自动驾驶车辆协同，都可以建模为多智能体强化学习（MARL）任务。QMIX是一种广受欢迎的MARL算法，已被用作Starcraft Multi-Agent Challenge（SMAC）、Difficulty-Enhanced Predator-Prey（DEPP）等基准环境的基线。近期QMIX的变体旨在放宽QMIX的单调性约束，从而在SMAC中获得性能提升。本文研究这些变体在代码层面的优化以及单调性约束。（1）我们发现这些变体的性能提升显著受到多种代码层面优化的影响。（2）实验结果表明，采用归一化优化的QMIX在SMAC中优于其他工作；（3）与这些工作的普遍看法不同，单调性约束能够提升SMAC和DEPP中的样本效率。我们还通过理论分析讨论了单调性约束为何在纯协作任务中表现良好。代码已开源于https://github.com/hijkzzz/pymarl2。
英文摘要：
Many complex multi-agent systems such as robot swarms control and autonomous vehicle coordination can be modeled as Multi-Agent Reinforcement Learning (MARL) tasks. QMIX, a widely popular MARL algorithm, has been used as a baseline for the benchmark environments, e.g., Starcraft Multi-Agent Challenge (SMAC), Difficulty-Enhanced Predator-Prey (DEPP). Recent variants of QMIX target relaxing the monotonicity constraint of QMIX, allowing for performance improvement in SMAC. In this paper, we investigate the code-level optimizations of these variants and the monotonicity constraint. (1) We find that such improvements of the variants are significantly affected by various code-level optimizations. (2) The experiment results show that QMIX with normalized optimizations outperforms other works in SMAC; (3) beyond the common wisdom from these works, the monotonicity constraint can improve sample efficiency in SMAC and DEPP. We also discuss why monotonicity constraints work well in purely cooperative tasks with a theoretical analysis. We open-source the code at \url{https://github.com/hijkzzz/pymarl2}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
