arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.02115cs.LGcs.DC
收藏

FedMoE-DA:基于域感知细粒度聚合的联邦混合专家模型

FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation

Sun Yat-Sen University, China(中山大学);The University of Hong Kong, Hong Kong(香港大学);Nanyang Technological University, Singapore(南洋理工大学);National University of Defense Technology, China(国防科技大学)

Ziwei Zhan, Wenkuan Zhao, Yuanqing Li, Weijie Liu, Xiaoxi Zhang, Chee Wei Tan, Chuan Wu, Deke Guo, Xu Chen

AI总结:

针对联邦学习中客户端资源受限和通信负担重的问题,提出基于混合专家架构的FedMoE-DA框架,采用域感知细粒度聚合和P2P通信,兼顾鲁棒性、个性化与通信效率。

摘要:

联邦学习(FL)是一种协作式机器学习方法,使多个客户端能够在无需共享私有数据的情况下训练模型。随着深度学习的兴起,大规模模型因其卓越的性能而受到广泛关注。然而,联邦学习中的一个关键挑战是计算和通信资源受限的客户端所带来的限制,这阻碍了这些大型模型的部署。混合专家(MoE)架构凭借其稀疏激活特性解决了这一挑战,该特性在推理和更新过程中减少了计算工作量和通信需求。此外,MoE通过允许每个专家专门处理数据分布的不同子集,促进了更好的个性化。为了减轻服务器与客户端之间的通信负担,我们提出了FedMoE-DA,一种新的联邦学习模型训练框架,该框架利用MoE架构,并结合一种新颖的域感知细粒度聚合策略,以同时增强鲁棒性、个性化和通信效率。具体而言,我们利用了客户端内专家模型与客户端间数据异构性之间的相关性。此外,我们利用客户端之间的点对点(P2P)通信进行选择性专家模型同步,从而显著减少服务器与客户端之间的传输。实验表明,我们的FedMoE-DA在降低服务器通信压力的同时,实现了优异的性能。

英文摘要:

Federated learning (FL) is a collaborative machine learning approach that enables multiple clients to train models without sharing their private data. With the rise of deep learning, large-scale models have garnered significant attention due to their exceptional performance. However, a key challenge in FL is the limitation imposed by clients with constrained computational and communication resources, which hampers the deployment of these large models. The Mixture of Experts (MoE) architecture addresses this challenge with its sparse activation property, which reduces computational workload and communication demands during inference and updates. Additionally, MoE facilitates better personalization by allowing each expert to specialize in different subsets of the data distribution. To alleviate the communication burdens between the server and clients, we propose FedMoE-DA, a new FL model training framework that leverages the MoE architecture and incorporates a novel domain-aware, fine-grained aggregation strategy to enhance the robustness, personalizability, and communication efficiency simultaneously. Specifically, the correlation between both intra-client expert models and inter-client data heterogeneity is exploited. Moreover, we utilize peer-to-peer (P2P) communication between clients for selective expert model synchronization, thus significantly reducing the server-client transmissions. Experiments demonstrate that our FedMoE-DA achieves excellent performance while reducing the communication pressure on the server.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑