arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.20441cs.MA
收藏

异步多智能体TD学习的有限时间分析

Finite-Time Analysis of Asynchronous Multi-Agent TD Learning

Nicolò Dal Fabbro, Arman Adibi, Aritra Mitra, George J. Pappas

AI总结:

本文首次证明在局部更新受异步延迟影响的MARL框架中仍可实现$N$倍线性收敛加速,并提供了异步多智能体TD学习算法\ exttt{AsyncMATD}的有限时间收敛分析。

摘要:

近期研究在理论上证明了多智能体强化学习(MARL)中合作的有益效果。在包含$N$个智能体的设置中,这种有益效果通常表现为$N$倍的线性收敛加速,即达到特定收敛精度所需的迭代次数减少与$N$成正比。本文首次证明,对于局部智能体更新受异步延迟影响的MARL框架,该加速特性同样成立。具体而言,我们考虑一个策略评估问题,其中多个智能体通过与中央聚合器通信来合作评估一个共同策略。在此设置下,我们研究了\ exttt{AsyncMATD}的有限时间收敛性,这是一种异步多智能体时间差分(TD)学习算法,其智能体的局部TD更新方向受异步有界延迟的影响。我们的主要贡献是提供了\ exttt{AsyncMATD}的有限时间分析,确立了其线性收敛加速,同时阐明了时变异步延迟对最终收敛速率的影响。

英文摘要:

Recent research endeavours have theoretically shown the beneficial effect of cooperation in multi-agent reinforcement learning (MARL). In a setting involving $N$ agents, this beneficial effect usually comes in the form of an $N$-fold linear convergence speedup, i.e., a reduction - proportional to $N$ - in the number of iterations required to reach a certain convergence precision. In this paper, we show for the first time that this speedup property also holds for a MARL framework subject to asynchronous delays in the local agents' updates. In particular, we consider a policy evaluation problem in which multiple agents cooperate to evaluate a common policy by communicating with a central aggregator. In this setting, we study the finite-time convergence of \texttt{AsyncMATD}, an asynchronous multi-agent temporal difference (TD) learning algorithm in which agents' local TD update directions are subject to asynchronous bounded delays. Our main contribution is providing a finite-time analysis of \texttt{AsyncMATD}, for which we establish a linear convergence speedup while highlighting the effect of time-varying asynchronous delays on the resulting convergence rate.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑