arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.12290cs.AI
收藏

多智能体强化学习中基于事实的智能体建模

Fact-based Agent modeling for Multi-Agent Reinforcement Learning

Hefei University of Technology(合肥工业大学)

Baofu Fang, Caiming Zheng, Hao Wang

AI总结:

针对未知场景下智能体建模的难题,提出基于事实的智能体建模方法FAM,仅利用局部信息通过变分自编码器重构其他智能体策略,有效提升了多智能体强化学习的策略学习效率与回报。

摘要:

在多智能体系统中,智能体需要在环境中与其他智能体进行交互和协作。智能体建模对于促进智能体交互并制定自适应合作策略至关重要。然而,在所有智能体策略同时被学习的非平稳环境中,智能体很难对其他智能体的信念、行为和意图进行建模。此外,现有方法通过行为克隆实现智能体建模,它们假设在执行或训练期间可以访问其他智能体的局部信息。然而,在以未知智能体为特征的未知场景中,由于隐私问题,如竞争团队、不可靠通信和联邦学习,这一假设是不可行的。为了消除这一假设并在未知场景中实现智能体建模,提出了基于事实的智能体建模(FAM)方法,其中基于事实的信念推断(FBI)网络仅基于其局部信息在部分可观测环境中对其他智能体进行建模。智能体采取行动后获得的奖励和观测被称为事实,FAM使用事实作为重构目标,通过变分自编码器学习其他智能体的策略表示。我们在各种多智能体粒子环境(MPE)上评估了FAM,并将结果与几种最先进的MARL算法进行了比较。实验结果表明,与基线方法相比,FAM能够通过在多智能体强化学习任务中制定自适应合作策略,有效提高智能体策略学习效率,同时在复杂的竞争-合作混合场景中获得更高的回报。

英文摘要:

In multi-agent systems, agents need to interact and collaborate with other agents in environments. Agent modeling is crucial to facilitate agent interactions and make adaptive cooperation strategies. However, it is challenging for agents to model the beliefs, behaviors, and intentions of other agents in non-stationary environment where all agent policies are learned simultaneously. In addition, the existing methods realize agent modeling through behavior cloning which assume that the local information of other agents can be accessed during execution or training. However, this assumption is infeasible in unknown scenarios characterized by unknown agents, such as competition teams, unreliable communication and federated learning due to privacy concerns. To eliminate this assumption and achieve agent modeling in unknown scenarios, Fact-based Agent modeling (FAM) method is proposed in which fact-based belief inference (FBI) network models other agents in partially observable environment only based on its local information. The reward and observation obtained by agents after taking actions are called facts, and FAM uses facts as reconstruction target to learn the policy representation of other agents through a variational autoencoder. We evaluate FAM on various Multiagent Particle Environment (MPE) and compare the results with several state-of-the-art MARL algorithms. Experimental results show that compared with baseline methods, FAM can effectively improve the efficiency of agent policy learning by making adaptive cooperation strategies in multi-agent reinforcement learning tasks, while achieving higher returns in complex competitive-cooperative mixed scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑