We study problems of federated control in Markov Decision Processes. To solve an MDP with large state space, multiple learning agents are introduced to collaboratively learn its optimal policy without communication of locally collected experience. In our settings, these agents have limited capabilities, which means they are restricted within different regions of the overall state space during the training process. In face of the difference among restricted regions, we firstly introduce concepts of leakage probabilities to understand how such heterogeneity affects the learning process, and then propose a novel communication protocol that we call Federated-Q protocol (FedQ), which periodically aggregates agents' knowledge of their restricted regions and accordingly modifies their learning problems for further training. In terms of theoretical analysis, we justify the correctness of FedQ as a communication protocol, then give a general result on sample complexity of derived algorithms FedQ-X with the RL oracle , and finally conduct a thorough study on the sample complexity of FedQ-SynQ. Specifically, FedQ-X has been shown to enjoy linear speedup in terms of sample complexity when workload is uniformly distributed among agents. Moreover, we carry out experiments in various environments to justify the efficiency of our methods.
Parameter-Free Federated TD Learning with Markov Noise in Heterogeneous Environments
Dept. of Computer Science and Automation, Indian Institute of Science（计算机科学与自动化系，印度科学研究院）；School of Electrical and Computer Engineering, Purdue University（电气与计算机工程学院，普渡大学）
Mirror Descent Policy Optimisation for Robust Constrained Markov Decision Processes
Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR)（前沿人工智能研究中心，科技研究局（A*STAR））；Singapore Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)（新加坡高性能计算研究所，科技研究局（A*STAR））；College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学，新加坡）
Remembering to Be Fair: Non-Markovian Fairness in Sequential Decision Making
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Schwartz Reisman Institute for Technology and Society（施瓦茨·赖斯曼技术与社会研究所）；University of Waterloo（滑铁卢大学）
Pursuing Overall Welfare in Federated Learning through Sequential Decision Making
Artificial Intelligence Graduate School, Ulsan National Institute of Science and Technology (UNIST)（乌山国立科学与技术研究所人工智能研究生院）；Policy Program, Seoul National University (SNU)（首尔国立大学政策项目）；Graduate School of Engineering Practice, Seoul National University (SNU)（首尔国立大学工程实践研究生院）；Institute of Engineering Research, Seoul National University (SNU)（首尔国立大学工程研究所）
