基于单跳邻居与计算落后节点缓解的分布式多智能体强化学习
Distributed Multi-Agent Reinforcement Learning with One-hop Neighbors and Compute Straggler Mitigation
AI总结：
本文提出DARL1N方法，通过将智能体信息交换限制在单跳邻居来降低多智能体强化学习的维度复杂度，并引入编码分布式学习架构缓解计算落后节点影响，在保持策略质量的同时显著提升训练效率和可扩展性。
摘要：
大多数多智能体强化学习（MARL）方法在处理问题的规模上存在局限。随着智能体数量的增加，由于联合状态空间和动作空间呈指数增长，找到最优行为所需的训练迭代次数也呈指数级上升。本文通过引入一种可扩展的MARL方法——基于单跳邻居的分布式多智能体强化学习（DARL1N）来应对这一局限。DARL1N是一种异策略（off-policy）演员-评论家（actor-critic）方法，通过在表示价值函数和策略函数时将智能体之间的信息交换限制在单跳邻居范围内，来解决维度灾难问题。每个智能体在单跳邻域上优化其价值函数和策略函数，显著降低了学习复杂度，同时通过在不同邻居数量和状态下进行训练来保持表达能力。这种结构使我们能够构建一个分布式学习框架，以进一步加快训练过程。然而，分布式计算系统中存在落后的计算节点，这些节点由于通信瓶颈、软件或硬件问题而运行缓慢或无响应。为了缓解这种有害的落后节点效应，我们引入了一种新颖的编码分布式学习架构，该架构利用编码理论提高学习系统对落后节点的鲁棒性。综合实验表明，DARL1N在不牺牲策略质量的情况下显著减少了训练时间，并且随着智能体数量的增加具有良好的可扩展性。此外，编码分布式学习架构在存在落后节点的情况下提高了训练效率。
英文摘要：
Most multi-agent reinforcement learning (MARL) methods are limited in the scale of problems they can handle. With increasing numbers of agents, the number of training iterations required to find the optimal behaviors increases exponentially due to the exponentially growing joint state and action spaces. This paper tackles this limitation by introducing a scalable MARL method called Distributed multi-Agent Reinforcement Learning with One-hop Neighbors (DARL1N). DARL1N is an off-policy actor-critic method that addresses the curse of dimensionality by restricting information exchanges among the agents to one-hop neighbors when representing value and policy functions. Each agent optimizes its value and policy functions over a one-hop neighborhood, significantly reducing the learning complexity, yet maintaining expressiveness by training with varying neighbor numbers and states. This structure allows us to formulate a distributed learning framework to further speed up the training procedure. Distributed computing systems, however, contain straggler compute nodes, which are slow or unresponsive due to communication bottlenecks, software or hardware problems. To mitigate the detrimental straggler effect, we introduce a novel coded distributed learning architecture, which leverages coding theory to improve the resilience of the learning system to stragglers. Comprehensive experiments show that DARL1N significantly reduces training time without sacrificing policy quality and is scalable as the number of agents increases. Moreover, the coded distributed learning architecture improves training efficiency in the presence of stragglers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
