arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.15935cs.LGcs.MA
收藏

通过新方法实现样本与通信高效的完全去中心化MARL策略评估:局部TD更新

Sample and Communication Efficient Fully Decentralized MARL Policy Evaluation via a New Approach: Local TD update

University of Wisconsin-Whitewater(威斯康星大学怀特沃特分校);North Carolina State University(北卡罗来纳州立大学);The Ohio State University(俄亥俄州立大学)

Fnu Hairi, Zifan Zhang, Jia Liu

AI总结:

针对完全去中心化MARL策略评估,本文提出局部TD更新方法,证明其在避免智能体漂移的同时能有效降低样本与通信复杂度至最优水平。

摘要:

在完全去中心化多智能体强化学习(MARL)的actor-critic框架中,关键组件之一是MARL策略评估(PE)问题。在该问题中,一组$N$个智能体通过与其邻居通信,协同评估给定策略下全局状态的价值函数。在MARL-PE中,一个关键挑战是如何降低样本和通信复杂度,其分别定义为收敛至某个$ε$-稳定点所需的训练样本数和通信轮数。为了降低MARL-PE中的通信复杂度,一种“自然”的想法是在每两次连续通信轮次之间执行多次局部TD更新步骤,以减少通信频率。然而,由于一般情况下智能体间奖励的异质性会导致潜在的“智能体漂移”现象,局部TD更新方法的有效性仍不明确。这引出了一个有趣且悬而未决的问题:局部TD更新方法能否带来较低的样本和通信复杂度?本文首次尝试回答这一基本问题。我们关注平均奖励下的MARL-PE设定,该设定受众多多智能体网络优化问题启发。我们的理论和实验结果证实,与基于共识的MARL-PE算法相比,允许执行多次局部TD更新步骤确实是降低MARL-PE样本和通信复杂度的有效方法。具体而言,为了收敛到MARL-PE的$ε$-稳定点,两次连续通信轮次之间的局部TD更新步骤数可以多达$\mathcal{O}(1/ε^{1/2}\log{(1/ε)})$。此外,我们从理论上证明,为了达到最优的样本复杂度,局部TD更新方法的通信复杂度为$\mathcal{O}(1/ε^{1/2}\log{(1/ε)})$。

英文摘要:

In actor-critic framework for fully decentralized multi-agent reinforcement learning (MARL), one of the key components is the MARL policy evaluation (PE) problem, where a set of $N$ agents work cooperatively to evaluate the value function of the global states for a given policy through communicating with their neighbors. In MARL-PE, a critical challenge is how to lower the sample and communication complexities, which are defined as the number of training samples and communication rounds needed to converge to some $ε$-stationary point. To lower communication complexity in MARL-PE, a "natural'' idea is to perform multiple local TD-update steps between each consecutive rounds of communication to reduce the communication frequency. However, the validity of the local TD-update approach remains unclear due to the potential "agent-drift'' phenomenon resulting from heterogeneous rewards across agents in general. This leads to an interesting open question: Can the local TD-update approach entail low sample and communication complexities? In this paper, we make the first attempt to answer this fundamental question. We focus on the setting of MARL-PE with average reward, which is motivated by many multi-agent network optimization problems. Our theoretical and experimental results confirm that allowing multiple local TD-update steps is indeed an effective approach in lowering the sample and communication complexities of MARL-PE compared to consensus-based MARL-PE algorithms. Specifically, the local TD-update steps between two consecutive communication rounds can be as large as $\mathcal{O}(1/ε^{1/2}\log{(1/ε)})$ in order to converge to an $ε$-stationary point of MARL-PE. Moreover, we show theoretically that in order to reach the optimal sample complexity, the communication complexity of local TD-update approach is $\mathcal{O}(1/ε^{1/2}\log{(1/ε)})$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑