arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.09038cs.LGcs.MAstat.ML
收藏

正规化!不要混合:没有显式集中结构的多智能体强化学习

Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures

University of Technology Sydney(悉尼科技大学);Faculty of Engineering and Information Technology(工程与信息技术学院)

Chapman Siu, Jason Traish, Richard Yi Da Xu

摘要:

我们建议使用正则化的多智能体强化学习,而不是学习显式的合作结构称为{ \em多智能体正则化Q学习}(MARQ)。许多MARL方法利用集中式结构,以便利用全局状态信息或在代理以分散方式行事时消除通信约束。而不是学习冗余的结构,在代理执行过程中被删除,我们建议,而不是利用共享的经验的代理,规范的个人政策,以促进结构化的探索。我们研究了几种不同的方法,如何MARQ可以显式或隐式地规范我们的政策,在多代理设置。MARQ旨在通过应用正则化约束来解决MARL上下文中的这些限制,这些约束可以纠正非政策分发代理经验中的偏差,并促进多样化的探索。我们的算法在几个基准多智能体环境中进行了评估,我们表明MARQ始终优于几个基线和最先进的算法;学习步骤更少,收敛到更高的回报。

英文摘要:

We propose using regularization for Multi-Agent Reinforcement Learning rather than learning explicit cooperative structures called {\em Multi-Agent Regularized Q-learning} (MARQ). Many MARL approaches leverage centralized structures in order to exploit global state information or removing communication constraints when the agents act in a decentralized manner. Instead of learning redundant structures which is removed during agent execution, we propose instead to leverage shared experiences of the agents to regularize the individual policies in order to promote structured exploration. We examine several different approaches to how MARQ can either explicitly or implicitly regularize our policies in a multi-agent setting. MARQ aims to address these limitations in the MARL context through applying regularization constraints which can correct bias in off-policy out-of-distribution agent experiences and promote diverse exploration. Our algorithm is evaluated on several benchmark multi-agent environments and we show that MARQ consistently outperforms several baselines and state-of-the-art algorithms; learning in fewer steps and converging to higher returns.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑