arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.18152cs.MA
收藏

合作多智能体强化学习的内在动作倾向一致性

Intrinsic Action Tendency Consistency for Cooperative Multi-Agent Reinforcement Learning

Junkai Zhang, Yifan Zhang, Xi Sheryl Zhang, Yifan Zang, Jian Cheng

AI总结:

针对CTDE范式下智能体动作倾向分歧影响训练效率的问题,提出Intrinsic Action Tendency Consistency方法,通过动作模型预测邻居动作倾向并计算合作内在奖励,结合RA-CTDE框架提升多智能体协作性能。

摘要:

在集中训练与分散执行(CTDE)范式下实现高效协作,仍是合作多智能体系统面临的挑战。我们发现智能体之间动作倾向的分歧是阻碍 CTDE 训练效率的重大障碍,需要大量训练样本才能就智能体策略达成统一共识。这种分歧源于 CTDE 信用分配过程中缺乏充分的团队共识相关指导信号。为解决此问题,我们提出了 Intrinsic Action Tendency Consistency,一种用于合作多智能体强化学习的新方法。它将通过动作模型获得的内在奖励整合到奖励加成的 CTDE(RA-CTDE)框架中。我们构建了一个动作模型,使周围智能体能够预测中心智能体的动作倾向。利用这些预测,我们计算出一个合作内在奖励,鼓励智能体使其动作与邻居的预测相匹配。我们通过理论分析建立了 RA-CTDE 与 CTDE 的等价性,证明可以使用智能体的个体目标实现 CTDE 的训练过程。基于这一见解,我们引入了一种结合内在奖励与 CTDE 的新方法。在 SMAC 和 GRF 基准的挑战性任务上的大量实验展示了我们方法性能的提升。

英文摘要:

Efficient collaboration in the centralized training with decentralized execution (CTDE) paradigm remains a challenge in cooperative multi-agent systems. We identify divergent action tendencies among agents as a significant obstacle to CTDE's training efficiency, requiring a large number of training samples to achieve a unified consensus on agents' policies. This divergence stems from the lack of adequate team consensus-related guidance signals during credit assignments in CTDE. To address this, we propose Intrinsic Action Tendency Consistency, a novel approach for cooperative multi-agent reinforcement learning. It integrates intrinsic rewards, obtained through an action model, into a reward-additive CTDE (RA-CTDE) framework. We formulate an action model that enables surrounding agents to predict the central agent's action tendency. Leveraging these predictions, we compute a cooperative intrinsic reward that encourages agents to match their actions with their neighbors' predictions. We establish the equivalence between RA-CTDE and CTDE through theoretical analyses, demonstrating that CTDE's training process can be achieved using agents' individual targets. Building on this insight, we introduce a novel method to combine intrinsic rewards and CTDE. Extensive experiments on challenging tasks in SMAC and GRF benchmarks showcase the improved performance of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑