异构联邦学习与分割语言模型
Heterogeneous Federated Learning with Splited Language Model
AI总结:
针对联邦分割学习在资源受限环境下的训练开销大和隐私风险问题,提出利用预训练图像Transformer(FedV)加速训练,并提出FedVZ通过零阶优化防御梯度反演攻击,在真实数据集上验证了有效性。
摘要:
联邦分割学习(FSL)是一种在实践中很有前景的分布式学习范式,它汇集了联邦学习(FL)和分割学习(SL)两种范式的优势,在确保模型隐私的同时,降低每个客户端的资源开销,尤其是在资源受限的环境(如物联网(IoT))中的大型Transformer模型上。然而,几乎所有工作都仅研究FSL中简单神经网络模型的性能。尽管有少量工作关注将视觉Transformer(ViT)作为模型架构,但它们从头训练ViT,从而导致在资源有限的设备上产生巨大的训练开销。因此,在本文中,我们利用预训练图像Transformer(PITs)作为初始模型,称为FedV,以加速训练过程并提高模型鲁棒性。此外,我们提出FedVZ来阻止梯度反演攻击,尤其具有与黑盒场景兼容的能力,在黑盒场景中梯度信息不可用。具体地,FedVZ通过利用零阶(ZO)优化来近似服务器梯度,该优化仅用一次前向过程代替反向传播。实验上,我们是首个在真实世界数据集、不同的部分设备参与和异构数据划分下对使用PITs的FSL方法进行系统评估的工作。我们的实验验证了我们算法的有效性。
英文摘要:
Federated Split Learning (FSL) is a promising distributed learning paradigm in practice, which gathers the strengths of both Federated Learning (FL) and Split Learning (SL) paradigms, to ensure model privacy while diminishing the resource overhead of each client, especially on large transformer models in a resource-constrained environment, e.g., Internet of Things (IoT). However, almost all works merely investigate the performance with simple neural network models in FSL. Despite the minor efforts focusing on incorporating Vision Transformers (ViT) as model architectures, they train ViT from scratch, thereby leading to enormous training overhead in each device with limited resources. Therefore, in this paper, we harness Pre-trained Image Transformers (PITs) as the initial model, coined FedV, to accelerate the training process and improve model robustness. Furthermore, we propose FedVZ to hinder the gradient inversion attack, especially having the capability compatible with black-box scenarios, where the gradient information is unavailable. Concretely, FedVZ approximates the server gradient by utilizing a zeroth-order (ZO) optimization, which replaces the backward propagation with just one forward process. Empirically, we are the first to provide a systematic evaluation of FSL methods with PITs in real-world datasets, different partial device participations, and heterogeneous data splits. Our experiments verify the effectiveness of our algorithms.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。