In actor-critic framework for fully decentralized multi-agent reinforcement learning (MARL), one of the key components is the MARL policy evaluation (PE) problem, where a set of $N$ agents work cooperatively to evaluate the value function of the global states for a given policy through communicating with their neighbors. In MARL-PE, a critical challenge is how to lower the sample and communication complexities, which are defined as the number of training samples and communication rounds needed to converge to some $ε$-stationary point. To lower communication complexity in MARL-PE, a "natural'' idea is to perform multiple local TD-update steps between each consecutive rounds of communication to reduce the communication frequency. However, the validity of the local TD-update approach remains unclear due to the potential "agent-drift'' phenomenon resulting from heterogeneous rewards across agents in general. This leads to an interesting open question: Can the local TD-update approach entail low sample and communication complexities? In this paper, we make the first attempt to answer this fundamental question. We focus on the setting of MARL-PE with average reward, which is motivated by many multi-agent network optimization problems. Our theoretical and experimental results confirm that allowing multiple local TD-update steps is indeed an effective approach in lowering the sample and communication complexities of MARL-PE compared to consensus-based MARL-PE algorithms. Specifically, the local TD-update steps between two consecutive communication rounds can be as large as $\mathcal{O}(1/ε^{1/2}\log{(1/ε)})$ in order to converge to an $ε$-stationary point of MARL-PE. Moreover, we show theoretically that in order to reach the optimal sample complexity, the communication complexity of local TD-update approach is $\mathcal{O}(1/ε^{1/2}\log{(1/ε)})$.
Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning
The Ohio State University, Columbus, Ohio, USA(俄亥俄州立大学);University of Wisconsin-Whitewater, Whitewater, Wisconsin, USA(威斯康星大学-威奇托分校);University of Colorado Boulder, Boulder, Colorado, USA(科罗拉多大学博尔德分校)
Finite-Sample Analysis For Decentralized Batch Multi-Agent Reinforcement Learning With Networked Agents
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Princeton University(普林斯顿大学);Northwestern University(西北大学);The Hong Kong University of Science and Technology(香港科技大学)
Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Framework
Washington University in St. Louis(华盛顿大学圣路易斯分校);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?
Zhejiang University(浙江大学);Zhejiang Provincial Engineering Research Center for Real-Time SmartTech in Urban Security Governance, School of Computer and Computing Science, Hangzhou City University(浙江省实时智慧城市治理实时智能技术工程研究中心,计算机与计算科学学院,杭州市大学);State Key Laboratory of Blockchain and Data Security, Zhejiang University(区块链与数据安全国家重点实验室,浙江大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江区)区块链与数据安全研究院)
MARLEM: A Multi-Agent Reinforcement Learning Simulation Framework for Implicit Cooperation in Decentralized Local Energy Markets
Department of Mechanical Engineering, Universidad de los Andes(机械工程系,洛斯安第斯大学);Department of Industrial Engineering, Universidad de los Andes(工业工程系,洛斯安第斯大学)
F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳市人工智能与机器人研究院);Tongji University(同济大学);Shanghai Research Institute for Intelligent Autonomous Systems(上海智能自主系统研究院);East China Normal University(华东师范大学);Shanghai Jiao Tong University(上海交通大学)
On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learning
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);DeepMind(深度思维);Massachusetts Institute of Technology(麻省理工学院)
多智能体强化学习(MARL)算法经常遭受指数样本复杂度对智能体数量的依赖,这种现象被称为\emph{the curse of multiagents}。在本文中,我们解决这一挑战,调查样本效率的无模型算法在\emph{decentralized} MARL,并旨在改善现有的算法沿这条线。对于学习(粗)一般和马尔可夫博弈中的相关均衡,我们提出了\emph{stage-based} V-学习算法,大大简化了最近的作品的算法设计和分析,并规避了一个相当复杂的无\emph{weighted}遗憾强盗子程序。针对马尔可夫势博弈中纳什均衡的学习问题,提出了一种独立的策略梯度算法,该算法采用分散的基于动量的方差缩减技术。我们所有的算法都是去中心化的,因为每个代理只能根据其本地信息做出决策。在学习过程中既不需要通信,也不需要集中协调,从而导致对大量代理的自然泛化。我们还提供了数值模拟来证实我们的理论研究结果。
Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning
Graduate School of Informatics, Kyoto University, Kyoto, Japan(京都大学信息学研究生院);Research Organization of Science and Technology, Ritsumeikan University, Shiga, Japan(立命馆大学科学技术研究机构)
Multi-agent In-context Coordination via Decentralized Memory Retrieval
National Key Laboratory of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Tencent(腾讯)
Learning Decentralized Traffic Signal Controllers with Multi-Agent Graph Reinforcement Learning
Northwestern Polytechnical University(西北工业大学);Harbin Engineering University(哈尔滨工程大学);Hong Kong University of Science and Technology(香港科技大学);Xidian University(西安电子科技大学);Nanyang Technological University(南洋理工大学)
Transferable Graphical MARL for Real-Time Estimation in Dynamic Wireless Networks
Department of Electrical and Computer Engineering, Rutgers University(罗格斯大学电气与计算机工程系);Department of Biostatistics and Bioinformatics, Duke University(杜克大学生物统计与生物信息学系);Department of Electrical and Systems Engineering, University of Pennsylvania(宾夕法尼亚大学电气与系统工程系)