arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2103.13147cs.LGcs.MAmath.OC
收藏

多智能体离策略TD学习:样本复杂度和通信复杂性接近最优的实时分析

Multi-Agent Off-Policy TD Learning: Finite-Time Analysis with Near-Optimal Sample Complexity and Communication Complexity

University of Utah(犹他大学)

Ziyi Chen, Yi Zhou, Rongrong Chen

摘要:

最近,离政策TD学习的有限时间收敛性得到了广泛的研究。然而,这种类型的收敛还没有很好地建立在多智能体设置,其中涵盖了更广泛的应用程序,从根本上更具挑战性的政策TD学习。针对马尔可夫抽样下的多智能体偏离策略TD学习问题,提出了两种分散式TD带校正(TDC)算法。特别是,我们的算法保持完全隐私的行动,政策和奖励的代理,并采用小批量采样,以减少采样方差和通信频率。在马尔可夫采样和线性函数近似下,证明了两种算法的有限时间采样复杂度都是$ε$量级,与集中式TD(0)和TDC的近似最优采样复杂度相匹配.重要的是,我们的算法的通信复杂性是在$\mathcal{O}(\ln ε^{-1})$的顺序,这是显着低于现有的分散TD(0)的通信复杂性度$\mathcal{O}(ε^{-1}\ln ε^{-1})$。实验证实了我们的理论发现。

英文摘要:

The finite-time convergence of off-policy TD learning has been comprehensively studied recently. However, such a type of convergence has not been well established for off-policy TD learning in the multi-agent setting, which covers broader applications and is fundamentally more challenging. This work develops two decentralized TD with correction (TDC) algorithms for multi-agent off-policy TD learning under Markovian sampling. In particular, our algorithms preserve full privacy of the actions, policies and rewards of the agents, and adopt mini-batch sampling to reduce the sampling variance and communication frequency. Under Markovian sampling and linear function approximation, we proved that the finite-time sample complexity of both algorithms for achieving an $ε$-accurate solution is in the order of $\mathcal{O}(ε^{-1}\ln ε^{-1})$, matching the near-optimal sample complexity of centralized TD(0) and TDC. Importantly, the communication complexity of our algorithms is in the order of $\mathcal{O}(\ln ε^{-1})$, which is significantly lower than the communication complexity $\mathcal{O}(ε^{-1}\ln ε^{-1})$ of the existing decentralized TD(0). Experiments corroborate our theoretical findings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑