arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.13841cs.LGcs.AI
收藏

联邦学习中多模型的公平并发训练

Fair Concurrent Training of Multiple Models in Federated Learning

Information Systems Technology and Design Pillar, Singapore University of Technology and Design(新加坡科技设计大学信息系统技术与设计学院);Electrical and Computer Engineering, Carnegie Mellon University(卡内基梅隆大学电子与计算机工程学院);Electrical and Computer Engineering, Northeastern University(东北大学电子与计算机工程学院)

Marie Siew, Haoran Zhang, Jong-Ik Park, Yuezhou Liu, Yichen Ruan, Lili Su, Stratis Ioannidis, Edmund Yeh, Carlee Joe-Wong

AI总结:

针对多模型联邦学习中任务异质难度及客户端参与意愿低导致的训练结果不公平问题,提出难度感知的动态分配算法FedFairMMFL与激励拍卖机制,在保证公平性与收敛性的同时提升任务训练效果。

摘要:

联邦学习(FL)支持跨多客户端的协同学习。在大多数联邦学习研究中,所有客户端都训练单个学习任务。然而,随着联邦学习应用的近期激增,可能会越来越需要同时训练多个联邦学习任务,共享客户端的计算与通信资源,我们将这一场景称为多模型联邦学习(Multiple-Model Federated Learning, MMFL)。\n当前的MMFL算法采用基于简单平均的客户端-任务分配方案,当联邦学习任务存在异质难度水平时(例如,模型更大的任务可能需要更多轮训练和更多数据),会导致性能不公平。正如将资源简单分配给具有异质资源需求的通用计算任务会产生不公平结果一样,将客户端简单分配给联邦学习任务也会引发不公平问题,表现为部分任务训练时间过长,或收敛精度更低。此外,在联邦学习场景中,由于客户端通常不会因训练贡献获得报酬,我们面临进一步挑战:部分客户端可能因计算成本过高而不愿训练某些任务,这可能加剧不同任务间训练结果的不公平性。\n为解决这两大挑战,我们首先设计了FedFairMMFL——一种感知任务难度的算法,可在每轮训练中为任务动态分配客户端。我们为该算法的公平性和收敛速率提供了理论保证。随后,我们提出了一种新颖的拍卖机制,用于激励客户端参与多任务训练,从而将客户端的训练算力公平分配到各任务中。我们分析了基于公平性的学习机制与激励机制对训练收敛的影响,最后在真实世界数据集上通过多组学习任务对所提算法进行了评估。

英文摘要:

Federated learning (FL) enables collaborative learning across multiple clients. In most FL work, all clients train a single learning task. However, the recent proliferation of FL applications may increasingly require multiple FL tasks to be trained simultaneously, sharing clients' computing and communication resources, which we call Multiple-Model Federated Learning (MMFL). Current MMFL algorithms use naive average-based client-task allocation schemes that can lead to unfair performance when FL tasks have heterogeneous difficulty levels, e.g., tasks with larger models may need more rounds and data to train. Just as naively allocating resources to generic computing jobs with heterogeneous resource needs can lead to unfair outcomes, naive allocation of clients to FL tasks can lead to unfairness, with some tasks having excessively long training times, or lower converged accuracies. Furthermore, in the FL setting, since clients are typically not paid for their training effort, we face a further challenge that some clients may not even be willing to train some tasks, e.g., due to high computational costs, which may exacerbate unfairness in training outcomes across tasks. We address both challenges by firstly designing FedFairMMFL, a difficulty-aware algorithm that dynamically allocates clients to tasks in each training round. We provide guarantees on airness and FedFairMMFL's convergence rate. We then propose a novel auction design that incentivizes clients to train multiple tasks, so as to fairly distribute clients' training efforts across the tasks. We show how our fairness-based learning and incentive mechanisms impact training convergence and finally evaluate our algorithm with multiple sets of learning tasks on real world datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑