arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.10697cs.LGstat.ML
收藏

联邦Q学习中异质性的裨益:线性加速及其他

The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyond

Carnegie Mellon University(卡内基梅隆大学)

Jiin Woo, Gauri Joshi, Yuejie Chi

AI总结:

本文研究联邦Q学习,针对无限视界表格型马尔可夫决策过程给出同步和异步变体的样本复杂度保证,提出带重要性平均的新型算法,实现鲁棒线性加速,揭示异质性在协作学习中的裨益。

摘要:

当强化学习(RL)所用数据由多个智能体以分布式方式收集时,联邦版RL算法可实现协作学习,且无需智能体共享其本地数据。本文研究联邦Q学习,该方法通过定期聚合仅在本地数据上训练的局部Q估计值,来学习最优Q函数。针对无限视界表格型马尔可夫决策过程,我们为联邦Q学习的同步和异步变体均提供了样本复杂度保证。在两种设置下,我们的界均表现出随智能体数量的线性加速,且与其他关键问题参数的依赖关系接近最优。\n在异步设置中,现有联邦Q学习分析采用局部Q估计值的等权平均,要求每个智能体都覆盖整个状态-动作空间。相比之下,我们改进后的样本复杂度与所有智能体的平均平稳状态-动作占用分布的最小项成反比,因此仅要求智能体共同覆盖整个状态-动作空间,揭示了异质性通过放宽单智能体场景的覆盖要求来赋能协作学习的裨益。然而,当局部轨迹高度异质时,其样本复杂度仍会受影响。对此,我们提出一种带重要性平均的新型联邦Q学习算法,为更频繁访问的状态-动作对赋予更大权重,无论局部行为策略的异质性如何,该算法都能实现鲁棒的线性加速,效果如同所有轨迹均被集中处理。

英文摘要:

When the data used for reinforcement learning (RL) are collected by multiple agents in a distributed manner, federated versions of RL algorithms allow collaborative learning without the need for agents to share their local data. In this paper, we consider federated Q-learning, which aims to learn an optimal Q-function by periodically aggregating local Q-estimates trained on local data alone. Focusing on infinite-horizon tabular Markov decision processes, we provide sample complexity guarantees for both the synchronous and asynchronous variants of federated Q-learning. In both cases, our bounds exhibit a linear speedup with respect to the number of agents and near-optimal dependencies on other salient problem parameters. In the asynchronous setting, existing analyses of federated Q-learning, which adopt an equally weighted averaging of local Q-estimates, require that every agent covers the entire state-action space. In contrast, our improved sample complexity scales inverse proportionally to the minimum entry of the average stationary state-action occupancy distribution of all agents, thus only requiring the agents to collectively cover the entire state-action space, unveiling the blessing of heterogeneity in enabling collaborative learning by relaxing the coverage requirement of the single-agent case. However, its sample complexity still suffers when the local trajectories are highly heterogeneous. In response, we propose a novel federated Q-learning algorithm with importance averaging, giving larger weights to more frequently visited state-action pairs, which achieves a robust linear speedup as if all trajectories are centrally processed, regardless of the heterogeneity of local behavior policies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑