Traditional distributed deep reinforcement learning (RL) commonly relies on exchanging the experience replay memory (RM) of each agent. Since the RM contains all state observations and action policy history, it may incur huge communication overhead while violating the privacy of each agent. Alternatively, this article presents a communication-efficient and privacy-preserving distributed RL framework, coined federated reinforcement distillation (FRD). In FRD, each agent exchanges its proxy experience replay memory (ProxRM), in which policies are locally averaged with respect to proxy states clustering actual states. To provide FRD design insights, we present ablation studies on the impact of ProxRM structures, neural network architectures, and communication intervals. Furthermore, we propose an improved version of FRD, coined mixup augmented FRD (MixFRD), in which ProxRM is interpolated using the mixup data augmentation algorithm. Simulations in a Cartpole environment validate the effectiveness of MixFRD in reducing the variance of mission completion time and communication cost, compared to the benchmark schemes, vanilla FRD, federated reinforcement learning (FRL), and policy distillation (PD).
Dynamic Proxy-Mixing: Transferring Replay Controllers from Small to Large Models for Continual Instruction Tuning
Department of Computer Science, Iowa State University（爱荷华州立大学计算机科学系）；Department of Computer Science, Kalinga Institute of Industrial Technology（卡林加工业技术学院计算机科学系）；Department of Civil, Construction & Environmental Engineering, Iowa State University（爱荷华州立大学土木、建筑与环境工程系）
In distributed reinforcement learning, it is common to exchange the experience memory of each agent and thereby collectively train their local models. The experience memory, however, contains all the preceding state observations and their corresponding policies of the host agent, which may violate the privacy of the agent. To avoid this problem, in this work, we propose a privacy-preserving distributed reinforcement learning (RL) framework, termed federated reinforcement distillation (FRD). The key idea is to exchange a proxy experience memory comprising a pre-arranged set of states and time-a…
Meta-reinforcement learning (meta-RL) has proven to be a successful framework for leveraging experience from prior tasks to rapidly learn new related tasks, however, current meta-RL approaches struggle to learn in sparse reward environments. Although existing meta-RL algorithms can learn strategies for adapting to new sparse reward tasks, the actual adaptation strategies are learned using hand-shaped reward functions, or require simple environments where random exploration is sufficient to encounter sparse reward. In this paper, we present a formulation of hindsight relabeling for meta-RL, whi…
Linear temporal logic (LTL) offers a simplified way of specifying tasks for policy optimization that may otherwise be difficult to describe with scalar reward functions. However, the standard RL framework can be too myopic to find maximally LTL satisfying policies. This paper makes two contributions. First, we develop a new value-function based proxy, using a technique we call eventual discounting, under which one can find policies that satisfy the LTL specification with highest achievable probability. Second, we develop a new experience replay method for generating off-policy data from on-pol…
This paper proposes a method for prioritizing the replay experience referred to as Hindsight Goal Ranking (HGR) in overcoming the limitation of Hindsight Experience Replay (HER) that generates hindsight goals based on uniform sampling. HGR samples with higher probability on the states visited in an episode with larger temporal difference (TD) error, which is considered as a proxy measure of the amount which the RL agent can learn from an experience. The actual sampling for large TD error is performed in two steps: first, an episode is sampled from the relay buffer according to the average TD e…
University of Catania（卡塔尼亚大学）；University of Turin（都灵大学）
In the medical field, multi-center collaborations are often sought to yield more generalizable findings by leveraging the heterogeneity of patient and clinical data. However, recent privacy regulations hinder the possibility to share data, and consequently, to come up with machine learning-based solutions that support diagnosis and prognosis. Federated learning (FL) aims at sidestepping this limitation by bringing AI-based solutions to data owners and only sharing local AI models, or parts thereof, that need then to be aggregated. However, most of the existing federated learning solutions are …
Khoury College of Computer Sciences（胡里计算机学院）；Northeastern University（东北大学）
Modern deep reinforcement learning methods have departed from the incremental learning required for eligibility traces, rendering the implementation of the $λ$-return difficult in this context. In particular, off-policy methods that utilize experience replay remain problematic because their random sampling of minibatches is not conducive to the efficient calculation of $λ$-returns. Yet replay-based methods are often the most sample efficient, and incorporating $λ$-returns into them is a viable way to achieve new state-of-the-art performance. Towards this, we propose the first method to enable …
HPCR: Holistic Proxy-based Contrastive Replay for Online Continual Learning
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Centre for Frontier AI Research, Institute of High Performance Computing, A*STAR（新加坡高性能计算研究院前沿人工智能研究中心（新加坡科技研究局））
Canonicalized Stable-List Replay for Private Federated Continual Learning over Language-Model Embeddings
Department of Computer Science, Iowa State University（爱荷华州立大学计算机科学系）；Department of Computer Science & Engineering, BRAC University（BRAC大学计算机科学与工程系）；Department of Civil, Construction & Environmental Engineering, Iowa State University（爱荷华州立大学土木、建设与环境工程系）
From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneity
School of Software, Shandong University（山东大学软件学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）
School of Operations Research and Information Engineering, Cornell University（康奈尔大学运营研究与信息工程学院）；Department of Industrial Systems Engineering and Management, National University of Singapore（新加坡国立大学工业系统工程与管理系）；SC Johnson College of Business, Cornell University（康奈尔大学SC约翰逊商学院）；Michael G. Foster School of Business, University of Washington（华盛顿大学迈克尔·G·福斯特商学院）
Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
The University of British Columbia（英属哥伦比亚大学）；Fudan University（复旦大学）；Royal College of Science, Imperial College London（伦敦帝国理工学院皇家科学学院）；Dyson School of Design Engineering（戴森设计工程学院）；Suzhou Institute of Biomedical Engineering and Technology (SIBET), Chinese Academy of Sciences（中国科学院苏州生物医学工程技术研究所）；East China Normal University（华东师范大学）
Class-wise Balancing Data Replay for Federated Class-Incremental Learning
School of Software, Shandong University（山东大学软件学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；Department of Electrical and Computer Engineering, University of British Columbia（不列颠哥伦比亚大学电气与计算机工程系）；Vector Institute（向量研究所）
Onboarding Without Forgetting: Hypernetwork Personalization with Data-Free Replay for Personalized Federated Learning
VinUni-Illinois Smart Health Center（VinUni-Illinois智能健康中心）；College of Engineering & Computer Science（工程与计算机科学学院）；University of Notre Dame（诺特丹大学）；Technische Universität Berlin（柏林技术大学）
CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning
Kyung Hee University（庆熙大学）；The University of Danang—Vietnam-Korea University of Information and Communication Technology（岘港大学—越南-韩国信息与通信技术大学）；Aalborg University（奥尔堡大学）
Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning
China University of Petroleum（中国石油大学）；WeBank（微众银行）；Pazhou Laboratory（琶洲实验室）；South China University of Technology（华南理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
ChainCaps: Composition-Safe Tool-Using Agents via Monotonic Capability Attenuation
Independent Researcher, Seattle, WA, USA（华盛顿州塞勒姆独立研究员）；Independent Researcher, New York City, NY, USA（纽约市纽约独立研究员）；King Abdullah University of Science and Technology（国王阿卜杜勒阿齐兹科学技术大学）
The Hong Kong University of Science and Technology（香港科技大学）；Microsoft Research Asia（微软亚洲研究院）
Non-IID data distribution across clients and poisoning attacks are two main challenges in real-world federated learning (FL) systems. While both of them have attracted great research interest with specific strategies developed, no known solution manages to address them in a unified framework. To universally overcome both challenges, we propose SmartFL, a generic approach that optimizes the server-side aggregation process with a small amount of proxy data collected by the service provider itself via a subspace training technique. Specifically, the aggregation weight of each participating client…
TrajSyn: Privacy-Preserving Dataset Distillation from Federated Model Trajectories for Server-Side Adversarial Training
Computer Science Department, Columbia University（哥伦比亚大学计算机科学系）；School of Computer Science and Engineering, Vellore Institute of Technology（维洛雷理工学院计算机科学与工程学院）；School of Information Technology, Deakin University（德肯大学信息科技学院）
Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Data
1 Embedded Systems Division, Silicon Austria Labs, Graz, Austria；2 Department of Computer Science, University of Innsbruck, Austria；5 Institute of Information Technology, University of Klagenfurt, Austria；3 TU-Graz SAL DES Lab, Silicon Austria Labs, Graz, Austria；4 Institute of Visual Computing, Graz University of Technology, Austria
