arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.03897cs.LGcs.DC
收藏

在异构环境中联邦Q学习的收敛速度研究

On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments

Northeastern University(东北大学);Tsinghua University(清华大学)

Leo Muxing Wang, Pengkun Yang, Lili Su

AI总结:

本文研究了在异构环境中同步联邦Q学习的收敛速度,发现K增加时误差下降线性加速,但E>1会导致性能下降,证明了误差衰减速度为Θ(E/T)。

摘要:

大规模多智能体系统常部署在广阔地理区域,其中智能体与异构环境交互。本文研究同步联邦Q学习,旨在通过K个智能体每E次迭代平均本地Q估计来学习最优Q函数。观察到K和E的收敛速度现象。与同构环境类似,K增加可线性减少采样随机性误差。但与同构设置相反,E>1导致显著性能下降。我们提供了环境异构性存在下的误差演变细粒度表征,误差随迭代次数T增加而衰减至零。E>1的缓慢收敛是根本而非分析误差。证明对于广泛步长范围,误差的∞范数无法比Θ(E/T)衰减更快。此外,实验表明收敛呈现两阶段现象。对于任何给定步长,存在明确的相变:初期误差快速衰减,但随后反弹并稳定。若能估计相变时间,为两阶段选择不同步长可加快整体收敛。

英文摘要:

Large-scale multi-agent systems are often deployed across wide geographic areas, where agents interact with heterogeneous environments. There is an emerging interest in understanding the role of heterogeneity in the performance of the federated versions of classic reinforcement learning algorithms. In this paper, we study synchronous federated Q-learning, which aims to learn an optimal Q-function by having $K$ agents average their local Q-estimates per $E$ iterations. We observe an interesting phenomenon on the convergence speeds in terms of $K$ and $E$. Similar to the homogeneous environment settings, there is a linear speed-up concerning $K$ in reducing the errors that arise from sampling randomness. Yet, in sharp contrast to the homogeneous settings, $E>1$ leads to significant performance degradation. Specifically, we provide a fine-grained characterization of the error evolution in the presence of environmental heterogeneity, which decay to zero as the number of iterations $T$ increases. The slow convergence of having $E>1$ turns out to be fundamental rather than an artifact of our analysis. We prove that, for a wide range of stepsizes, the $\ell_{\infty}$ norm of the error cannot decay faster than $Θ(E/T)$. In addition, our experiments demonstrate that the convergence exhibits an interesting two-phase phenomenon. For any given stepsize, there is a sharp phase-transition of the convergence: the error decays rapidly in the beginning yet later bounces up and stabilizes. Provided that the phase-transition time can be estimated, choosing different stepsizes for the two phases leads to faster overall convergence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑