arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.01654cs.LGcs.SYeess.SYmath.OCstat.ML
收藏

多智能体自然演员-评论家强化学习算法

Multi-agent Natural Actor-critic Reinforcement Learning Algorithms

Indian Institute of Technology Bombay(印度理工学院孟买分校)

Prashant Trivedi, Nandyala Hemachandra

摘要:

多智能体行动者-批评者算法是强化学习范式的重要组成部分。在这项工作中,我们提出了三个完全分散的多智能体自然演员-评论家(MAN)算法。我们的目标是共同找到一个联合政策,最大限度地提高这些代理人的平均长期回报。在没有中央控制器的情况下,为了保护隐私,代理通过时变通信网络向其邻居传达一些信息。我们证明了所有3 MAN算法的收敛到一个全局渐近稳定的ODE集对应的演员更新,这些使用线性函数近似。我们证明了连续迭代策略之间的Kullback-Leibler发散与目标函数的梯度成正比。我们观察到,Fisher信息矩阵的最小奇异值是在政策参数维度的倒数范围内。使用这一点,我们从理论上表明,MAN算法的确定性变体的最佳值在每个节点占主导地位的标准的基于梯度的多智能体演员-评论家(MAAC)算法。据我们所知,这是多智能体强化学习(MARL)的第一个这样的结果。为了说明我们提出的算法的有用性,我们实现了他们的双车道交通网络,以减少平均网络拥塞。我们观察到一个几乎25%的平均拥塞减少2 MAN算法,在另一个MAN算法的平均拥塞是与MAAC算法。我们还考虑了一个通用的$15$代理MARL的MAN算法的性能是一样好的MAAC算法。

英文摘要:

Multi-agent actor-critic algorithms are an important part of the Reinforcement Learning paradigm. We propose three fully decentralized multi-agent natural actor-critic (MAN) algorithms in this work. The objective is to collectively find a joint policy that maximizes the average long-term return of these agents. In the absence of a central controller and to preserve privacy, agents communicate some information to their neighbors via a time-varying communication network. We prove convergence of all the 3 MAN algorithms to a globally asymptotically stable set of the ODE corresponding to actor update; these use linear function approximations. We show that the Kullback-Leibler divergence between policies of successive iterates is proportional to the objective function's gradient. We observe that the minimum singular value of the Fisher information matrix is well within the reciprocal of the policy parameter dimension. Using this, we theoretically show that the optimal value of the deterministic variant of the MAN algorithm at each iterate dominates that of the standard gradient-based multi-agent actor-critic (MAAC) algorithm. To our knowledge, it is a first such result in multi-agent reinforcement learning (MARL). To illustrate the usefulness of our proposed algorithms, we implement them on a bi-lane traffic network to reduce the average network congestion. We observe an almost 25\% reduction in the average congestion in 2 MAN algorithms; the average congestion in another MAN algorithm is on par with the MAAC algorithm. We also consider a generic $15$ agent MARL; the performance of the MAN algorithms is again as good as the MAAC algorithm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑