arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.02731cs.LGmath.OC
收藏

平均场多智能体强化学习:一种分散网络方法

Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach

University of California, Berkeley(加州大学伯克利分校);Tsinghua-Berkeley Shenzhen Institute(清华-伯克利深圳学院);University of Southern California(南加州大学)

Haotian Gu, Xin Guo, Xiaoli Wei, Renyuan Xu

摘要:

多智能体强化学习(MARL)面临的挑战之一是为大型系统设计有效的学习算法,在该系统中每个智能体仅具有整个系统的有限或部分信息。虽然在分析具有社交网络和团队视频游戏的代理网络的去中心化MARL方面取得了令人兴奋的进展,但理论上对具有用于建模自动驾驶车辆,乘车共享以及数据和交通路由的状态网络的去中心化MARL知之甚少。本文提出了一个局部训练和分散执行的框架来研究具有状态网络的MARL。局部化训练意味着Agent只需要在训练阶段收集邻近状态的局部信息;分散化执行意味着Agent可以在训练之后执行学习到的分散化策略,这些策略仅取决于Agent的当前状态。理论分析由三个关键部分组成:第一是将MARL系统重新表述为具有多个代理团队的网络化马尔可夫决策过程,使得能够以局部方式更新相关团队Q函数;第二是用于价值函数的Bellman方程和概率测度空间上的适当Q函数;第三是团队Q函数的指数衰减特性,使其具有有效的样本效率和可控的误差近似。理论分析为一种新的算法LTDE-Neural-AC铺平了道路,其中提出了过参数化神经网络的Actor-Critic方法。的收敛性和样本的复杂性的建立和证明是可扩展的代理和状态的大小。据我们所知,这是第一个基于神经网络的MARL算法的网络结构和可证明的收敛保证。

英文摘要:

One of the challenges for multi-agent reinforcement learning (MARL) is designing efficient learning algorithms for a large system in which each agent has only limited or partial information of the entire system. While exciting progress has been made to analyze decentralized MARL with the network of agents for social networks and team video games, little is known theoretically for decentralized MARL with the network of states for modeling self-driving vehicles, ride-sharing, and data and traffic routing. This paper proposes a framework of localized training and decentralized execution to study MARL with network of states. Localized training means that agents only need to collect local information in their neighboring states during the training phase; decentralized execution implies that agents can execute afterwards the learned decentralized policies, which depend only on agents' current states. The theoretical analysis consists of three key components: the first is the reformulation of the MARL system as a networked Markov decision process with teams of agents, enabling updating the associated team Q-function in a localized fashion; the second is the Bellman equation for the value function and the appropriate Q-function on the probability measure space; and the third is the exponential decay property of the team Q-function, facilitating its approximation with efficient sample efficiency and controllable error. The theoretical analysis paves the way for a new algorithm LTDE-Neural-AC, where the actor-critic approach with over-parameterized neural networks is proposed. The convergence and sample complexity is established and shown to be scalable with respect to the sizes of both agents and states. To the best of our knowledge, this is the first neural network based MARL algorithm with network structure and provably convergence guarantee.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑