arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.19600cs.LG
收藏

自适应自蒸馏用于最小化异构联邦学习中的客户端漂移

Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning

Indian Institute of Science(印度科学研究院);Indian Institute of Technology (IIT) Roorkee(印度理工学院(IIT)拉尔基)

M Yashwanth, Gaurav Kumar Nayak, Arya Singh, Yogesh Simmhan, Anirban Chakraborty

AI总结:

本文提出自适应自蒸馏技术,用于减少异构联邦学习中的客户端漂移问题,通过自适应调整正则化参数提升模型性能。

摘要:

联邦学习(FL)是一种机器学习范式,使客户端能够通过聚合本地训练的模型共同训练一个全局模型,而无需共享任何本地训练数据。在实践中,每个客户端观察到的本地数据分布往往存在显著的异质性(例如类别不平衡)。在这样的非iid标签分布下,FL会受到'客户端漂移'问题的影响,即每个客户端都会漂向自己的局部最优。这导致了收敛速度变慢和聚合模型性能不佳。为了解决这一限制,我们提出了一种基于自适应自蒸馏(ASD)的新型正则化技术,用于在客户端侧训练模型。我们的正则化方案根据全局模型的预测熵和客户端数据标签分布,自适应地调整到每个客户端的训练数据。本文表明,我们提出的正则化(ASD)可以轻松地集成到现有的、最先进的FL算法之上,从而进一步提升这些现成方法的性能。我们理论解释了ASD正则化器的引入如何减少客户端漂移,并通过实验证明训练模型的泛化能力。我们通过广泛的实验在多个现实世界基准上展示了我们方法的有效性,并展示了当所提出的正则器与流行FL方法结合时,性能有显著提升。

英文摘要:

Federated Learning (FL) is a machine learning paradigm that enables clients to jointly train a global model by aggregating the locally trained models without sharing any local training data. In practice, there can often be substantial heterogeneity (e.g., class imbalance) across the local data distributions observed by each of these clients. Under such non-iid label distributions across clients, FL suffers from the 'client-drift' problem where every client drifts to its own local optimum. This results in slower convergence and poor performance of the aggregated model. To address this limitation, we propose a novel regularization technique based on adaptive self-distillation (ASD) for training models on the client side. Our regularization scheme adaptively adjusts to each client's training data based on the global model's prediction entropy and the client-data label distribution. We show in this paper that our proposed regularization (ASD) can be easily integrated atop existing, state-of-the-art FL algorithms, leading to a further boost in the performance of these off-the-shelf methods. We theoretically explain how incorporation of ASD regularizer leads to reduction in client-drift and empirically justify the generalization ability of the trained model. We demonstrate the efficacy of our approach through extensive experiments on multiple real-world benchmarks and show substantial gains in performance when the proposed regularizer is combined with popular FL methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑