arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.09646cs.MAcs.AIcs.LG
收藏

RPM:多智能体强化学习的可推广行为

RPM: Generalizable Behaviors for Multi-Agent Reinforcement Learning

Nanyang Technological University(南洋理工大学);Sea AI Lab(Sea人工智能实验室)

Wei Qiu, Xiao Ma, Bo An, Svetlana Obraztsova, Shuicheng Yan, Zhongwen Xu

摘要:

尽管最近在多智能体强化学习(MARL)方面取得了进展,但MARL智能体很容易过拟合训练环境,并且在其他智能体行为不同的评估场景中表现不佳。因此,获得MARL代理的可推广的政策是必要的,但主要是由于复杂的多代理交互的挑战。在这项工作中,我们用马尔可夫博弈模型的问题,并提出了一个简单而有效的方法,排名策略记忆(RPM),收集不同的多智能体轨迹训练MARL政策具有良好的泛化能力。RPM的主要思想是维护策略的查找存储器。特别是,我们试图通过对训练集回报进行排名来保存策略,从而获得各种级别的行为,即,训练环境中代理的事件返回;当事件开始时,学习代理可以从RPM中选择一个策略作为行为策略。这种创新的自我发挥训练框架利用代理的过去的政策,并保证了训练数据中的多代理交互的多样性。我们在MARL算法的基础上实现RPM,并在Melting Pot上进行了大量的实验。事实证明,RPM使MARL代理能够在多代理泛化评估场景中与看不见的代理进行交互并完成给定的任务,并且它可以显着提高性能,平均可达402%。

英文摘要:

Despite the recent advancement in multi-agent reinforcement learning (MARL), the MARL agents easily overfit the training environment and perform poorly in the evaluation scenarios where other agents behave differently. Obtaining generalizable policies for MARL agents is thus necessary but challenging mainly due to complex multi-agent interactions. In this work, we model the problem with Markov Games and propose a simple yet effective method, ranked policy memory (RPM), to collect diverse multi-agent trajectories for training MARL policies with good generalizability. The main idea of RPM is to maintain a look-up memory of policies. In particular, we try to acquire various levels of behaviors by saving policies via ranking the training episode return, i.e., the episode return of agents in the training environment; when an episode starts, the learning agent can then choose a policy from the RPM as the behavior policy. This innovative self-play training framework leverages agents' past policies and guarantees the diversity of multi-agent interaction in the training data. We implement RPM on top of MARL algorithms and conduct extensive experiments on Melting Pot. It has been demonstrated that RPM enables MARL agents to interact with unseen agents in multi-agent generalization evaluation scenarios and complete given tasks, and it significantly boosts the performance up to 402% on average.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑