多智能体协调的双层Actor-Critic模型
Bi-level Actor-Critic for Multi-agent Coordination
摘要：
协调是多智能体系统中的基本问题之一。典型的多智能体强化学习（MARL）方法平等地对待智能体，目标是当存在多个均衡时将马尔可夫博弈求解到任意纳什均衡（NE），因此缺乏NE选择的解决方案。在本文中，我们处理代理\emph{unequally}和考虑Stackelberg均衡作为一个潜在的更好的收敛点比纳什均衡的帕累托优势，特别是在合作环境。在马尔可夫博弈下，我们正式定义了寻找Stackelberg均衡的双层强化学习问题。我们提出了一种新的双层演员批评学习方法，允许代理人有不同的知识库（因此智能），而他们的行动仍然可以同时和分布式执行。收敛性证明，而由此产生的学习算法进行测试，对国家的艺术。我们发现，建议的双层演员-评论家算法成功地收敛到Stackelberg平衡矩阵游戏，并找到一个不对称的解决方案，在高速公路合并环境。
英文摘要：
Coordination is one of the essential problems in multi-agent systems. Typically multi-agent reinforcement learning (MARL) methods treat agents equally and the goal is to solve the Markov game to an arbitrary Nash equilibrium (NE) when multiple equilibra exist, thus lacking a solution for NE selection. In this paper, we treat agents \emph{unequally} and consider Stackelberg equilibrium as a potentially better convergence point than Nash equilibrium in terms of Pareto superiority, especially in cooperative environments. Under Markov games, we formally define the bi-level reinforcement learning problem in finding Stackelberg equilibrium. We propose a novel bi-level actor-critic learning method that allows agents to have different knowledge base (thus intelligent), while their actions still can be executed simultaneously and distributedly. The convergence proof is given, while the resulting learning algorithm is tested against the state of the arts. We found that the proposed bi-level actor-critic algorithm successfully converged to the Stackelberg equilibria in matrix games and find an asymmetric solution in a highway merge environment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
