arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.16981cs.LGmath.OCstat.ML
收藏

联邦Q学习中的样本-通信复杂度权衡

The Sample-Communication Complexity Trade-off in Federated Q-Learning

Carnegie Mellon University(卡内基梅隆大学)

Sudeep Salgia, Yuejie Chi

AI总结:

该研究针对联邦Q学习的样本与通信复杂度权衡问题,证明了间歇通信算法的逆下界,并提出Fed-DVR-Q算法首次实现阶最优的双复杂度,完整刻画了二者的权衡关系。

摘要:

我们研究联邦Q学习问题,其中M个智能体旨在协作学习状态空间与动作空间均有限的未知无限时域马尔可夫决策过程的最优Q函数。我们针对广泛使用的间歇通信类算法,探究样本复杂度与通信复杂度之间的权衡关系。我们首先建立了逆结果:若某联邦Q学习算法能在单智能体样本复杂度上随智能体数量实现任意加速,则其通信成本至少需达到1/(1-γ)的量级(对数因子可忽略),其中γ为折扣因子。我们还提出了名为Fed-DVR-Q的新算法,这是首个同时达到阶最优样本复杂度与通信复杂度的联邦Q学习算法。综上,这些结果完整刻画了联邦Q学习中的样本-通信复杂度权衡特性。

英文摘要:

We consider the problem of federated Q-learning, where $M$ agents aim to collaboratively learn the optimal Q-function of an unknown infinite-horizon Markov decision process with finite state and action spaces. We investigate the trade-off between sample and communication complexities for the widely used class of intermittent communication algorithms. We first establish the converse result, where it is shown that a federated Q-learning algorithm that offers any speedup with respect to the number of agents in the per-agent sample complexity needs to incur a communication cost of at least an order of $\frac{1}{1-γ}$ up to logarithmic factors, where $γ$ is the discount factor. We also propose a new algorithm, called Fed-DVR-Q, which is the first federated Q-learning algorithm to simultaneously achieve order-optimal sample and communication complexities. Thus, together these results provide a complete characterization of the sample-communication complexity trade-off in federated Q-learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑