arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.19477cs.MAcs.LG
收藏

基于信念图辅助训练的多智能体合作博弈

Multi-agent Cooperative Games Using Belief Map Assisted Training

Syracuse University（雪城大学）；Fayetteville-Manlius High School（费耶特维尔-曼利厄斯高中）；Air Force Research Laboratory（空军研究实验室）；Duke University（杜克大学）

Qinwei Huang, Chen Luo, Alex B. Wu, Simon Khan, Hai Li, Qinru Qiu

发表信息：
ECAI 2023. IOS Press, 2023: 1617-1624
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出信念图辅助多智能体系统BAMS，利用神经符号信念图提供额外一致反馈以优化消息传递网络，在合作捕食者与猎物博弈中训练轮数减少66%、完成步数减少34.62%。

摘要：

在多智能体系统中，智能体通过消息传递系统共享局部观测，以获得用于决策和协作的全局态势感知。何时发送消息、如何编码消息以及如何利用接收到的消息，直接影响智能体之间协作的有效性。在使用强化学习（RL）训练多智能体合作博弈时，消息传递系统需要与智能体策略一起优化。这随之增加了模型的复杂性，并对学习的收敛性和性能提出了重大挑战。为了解决这一问题，我们提出了信念图辅助多智能体系统（BAMS），该系统利用神经符号信念图来增强训练。信念图对智能体的隐藏状态进行解码，以提供智能体对环境理解及其他智能体状态的符号化表示。符号化表示的简洁性使得能够收集真实信息并与信念进行比较，从而为学习提供额外的反馈通道。与强化学习中来自奖励的零散且延迟的反馈相比，来自信念图的反馈更加一致和可靠。使用BAMS的智能体能够学习到更有效的消息传递网络，从而更好地相互理解，进而在具有不同地图复杂度的合作捕食者与猎物博弈中取得更优表现，并与以往的多智能体消息传递模型进行比较。仿真结果表明，BAMS将训练轮数减少了66%，并且应用BAMS模型的智能体完成博弈的平均步数减少了34.62%。

英文摘要：

In a multi-agent system, agents share their local observations to gain global situational awareness for decision making and collaboration using a message passing system. When to send a message, how to encode a message, and how to leverage the received messages directly affect the effectiveness of the collaboration among agents. When training a multi-agent cooperative game using reinforcement learning (RL), the message passing system needs to be optimized together with the agent policies. This consequently increases the model's complexity and poses significant challenges to the convergence and performance of learning. To address this issue, we propose the Belief-map Assisted Multi-agent System (BAMS), which leverages a neuro-symbolic belief map to enhance training. The belief map decodes the agent's hidden state to provide a symbolic representation of the agent's understanding of the environment and other agent's status. The simplicity of symbolic representation allows the gathering and comparison of the ground truth information with the belief, which provides an additional channel of feedback for the learning. Compared to the sporadic and delayed feedback coming from the reward in RL, the feedback from the belief map is more consistent and reliable. Agents using BAMS can learn a more effective message passing network to better understand each other, resulting in better performance in a cooperative predator and prey game with varying levels of map complexity and compare it to previous multi-agent message passing models. The simulation results showed that BAMS reduced training epochs by 66\%, and agents who apply the BAMS model completed the game with 34.62\% fewer steps on average.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑