arXiv 2405.093942025-06-22cs.LGcs.DC

SPD-CFL：面向高效持续联邦学习的逐步参数丢弃

SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learning

State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（网络与交换技术国家重点实验室，北京邮电大学）；College of Computing and Data Science, Nanyang Technological University（computing and Data Science学院，南洋理工大学）；UCL Cancer Institute, University College London（伦敦大学学院癌症研究所）；Department of Broadband Communication, Peng Cheng Laboratory（宽带通信部，鹏城实验室）

Yuning Yang, Han Yu, Chuan Sun, Tianrun Gao, Xiaohong Liu, Xiaodong Xu, Ping Zhang, Guangyu Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对联邦学习通信开销高和参数丢弃率需人工试错的问题，提出SPD-CFL方法，通过逐步丢弃参数和持续学习，在指定性能下自动确定丢弃率，显著提升AUC并降低通信开销。

摘要：

联邦学习（FL）是一种协作式机器学习范式，用于在隐私保护下对本地敏感数据进行模型训练。基于预训练的Transformer模型已成为广泛下游任务微调的有用基础模型（FM）。然而，大规模预训练模型使得传统FL在资源受限的物联网中面临挑战，因为通信开销过高。这激发了参数高效微调（PEFT）研究领域。现有PEFT方法试图在给定丢弃水平下优化模型性能。这种方法将负担置于人类用户身上，需要通过反复试验找到能提供满意性能水平的丢弃率，这既耗时又耗费资源。为解决这一局限，我们提出了逐步参数丢弃的持续联邦学习方法（SPD-CFL）。它不是预先定义期望的丢弃率，而是允许用户指定目标性能水平，然后尝试为给定FL模型找到最合适的丢弃率。具体而言，在服务器端，SPD-CFL通过降低低秩适应（LoRA）的秩来逐步丢弃可训练参数，以提高通信效率。设计了基于敏感性的梯度一致性（SGC）度量，以促进参数丢弃的自适应调整。此外，SPD-CFL在客户端引入持续学习（CL），以缓解在异构FL下因不同参数丢弃率导致的不一致最优而造成的性能下降。在公共基准数据集CIFAR-10和真实世界医学面部数据集上的大量实验表明，SPD-CFL显著优于最先进方法。与最佳性能基线相比，它实现了2.07%更高的测试AUC，同时将通信开销降低了29.53%。

英文摘要：

Federated Learning (FL) is a collaborative machine learning paradigm for training models on local sensitive data with privacy protection. Pre-trained transformer-based models have emerged as useful foundation models (FMs) to be fine-tuned for a wide range of downstream tasks. However, large-scale pre-trained models make it challenging for traditional FL due to high communication overhead in the resource-constrained IoT. This has inspired the field of parameter-efficient fine-tuning (PEFT) research. Existing PEFT methods attempt to optimize model performance at the given dropout level. Such an approach places the burden on human users to find a dropout rate that provides a satisfactory level of performance through trial-and-error, which is time consuming and resource intensive. To address this limitation, we propose the Step-wise Parameter Dropout for Continual Federated Learning (SPD-CFL) approach. Instead of pre-defining a desired dropout rate, it allows users to specify the target level of performance and then attempts to find the most suitable dropout rate for the given FL model. Specifically, on the server side, SPD-CFL drops trainable parameters in a stepwise manner to improve communication efficiency by reducing the rank of low-rank adaptation (LoRA). The sensitivity-based gradient consistency (SGC) measure is designed to facilitate the adaptive adjustment of parameter dropout. In addition, SPD-CFL introduces continual learning (CL) on the client side to mitigate performance degradation due to the inconsistent optima with distinct parameter dropout rates under heterogeneous FL. Extensive experiments on the public benchmark dataset CIFAR-10 and a real-world medical Face dataset demonstrate significant superiority of SPD-CFL over state-of-the-art methods. Compared to the best-performing baseline, it achieves a 2.07% higher test AUC while reducing communication overhead by 29.53%.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。