协作多智能体强化学习中个体全局最大值的再思考
Rethinking Individual Global Max in Cooperative Multi-Agent Reinforcement Learning
摘要：
在协作式多智能体强化学习中，集中训练和分散执行（CTDE）取得了显着的成功。个体全局最大值分解是CTDE的重要组成部分，它度量了局部策略和联合策略之间的一致性。大多数基于IGM的研究集中在如何建立这种一致的关系，但很少注意到检查IGM的潜在缺陷。在这项工作中，我们揭示了IGM条件是一个有损分解，有损分解的误差将积累在基于超网络的方法。为了解决上述问题，我们建议采用一种模仿学习策略，将有损分解与Bellman迭代分离，从而避免误差累积。该策略在零视角的星际争霸多智能体挑战基准问题上得到了理论证明和实验验证。结果还证实，所提出的方法优于最先进的基于IGM的方法。
英文摘要：
In cooperative multi-agent reinforcement learning, centralized training and decentralized execution (CTDE) has achieved remarkable success. Individual Global Max (IGM) decomposition, which is an important element of CTDE, measures the consistency between local and joint policies. The majority of IGM-based research focuses on how to establish this consistent relationship, but little attention has been paid to examining IGM's potential flaws. In this work, we reveal that the IGM condition is a lossy decomposition, and the error of lossy decomposition will accumulated in hypernetwork-based methods. To address the above issue, we propose to adopt an imitation learning strategy to separate the lossy decomposition from Bellman iterations, thereby avoiding error accumulation. The proposed strategy is theoretically proved and empirically verified on the StarCraft Multi-Agent Challenge benchmark problem with zero sight view. The results also confirm that the proposed method outperforms state-of-the-art IGM-based approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
