When the data used for reinforcement learning (RL) are collected by multiple agents in a distributed manner, federated versions of RL algorithms allow collaborative learning without the need for agents to share their local data. In this paper, we consider federated Q-learning, which aims to learn an optimal Q-function by periodically aggregating local Q-estimates trained on local data alone. Focusing on infinite-horizon tabular Markov decision processes, we provide sample complexity guarantees for both the synchronous and asynchronous variants of federated Q-learning. In both cases, our bounds exhibit a linear speedup with respect to the number of agents and near-optimal dependencies on other salient problem parameters.
In the asynchronous setting, existing analyses of federated Q-learning, which adopt an equally weighted averaging of local Q-estimates, require that every agent covers the entire state-action space. In contrast, our improved sample complexity scales inverse proportionally to the minimum entry of the average stationary state-action occupancy distribution of all agents, thus only requiring the agents to collectively cover the entire state-action space, unveiling the blessing of heterogeneity in enabling collaborative learning by relaxing the coverage requirement of the single-agent case. However, its sample complexity still suffers when the local trajectories are highly heterogeneous. In response, we propose a novel federated Q-learning algorithm with importance averaging, giving larger weights to more frequently visited state-action pairs, which achieves a robust linear speedup as if all trajectories are centrally processed, regardless of the heterogeneity of local behavior policies.
Best of Both Worlds in Federated LSA: Speedup When Possible, Personalization Always
CNRS(法国国家科学研究中心);École Polytechnique(巴黎综合理工学院);Institut Polytechnique de Paris(巴黎理工学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);EPITA(高等计算机与技术学院)
Achieving Linear Speedup with ProxSkip in Distributed Stochastic Optimization
School of Computer Science and Engineering, Suzhou University of Technology(苏州科技大学计算机科学与工程学院);Department of Electrical Engineering, Kuwait University(科威特大学电子工程系);Center for Machine Learning Research, Peking University(北京大学机器学习研究中心);King Abdullah University of Science and Technology(卡塔尔科学与技术大学);School of Mathematics, Southeast University(东南大学数学系);Purple Mountain Laboratories(紫金山实验室)
SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);Meituan(美团);University of Chinese Academy of Sciences(中国科学院大学)
FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models
School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院);Medical College, Tianjin University(天津大学医学院);School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院通用人工智能国家重点实验室);Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China(广州黄埔 Pazhou 实验室)
Federated Learning with Manifold Regularization and Normalized Update Reaggregation
School of Information and Electronics, Beijing Institute of Technology(北京理工大学信息与电子学院);JD Explore Academy(京东探索研究院);School of Computer Science, Wuhan University(武汉大学计算机学院)
CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning
Kyung Hee University(庆熙大学);The University of Danang—Vietnam-Korea University of Information and Communication Technology(岘港大学—越南-韩国信息与通信技术大学);Aalborg University(奥尔堡大学)