Decentralized learning (DL) leverages edge devices for collaborative model training while avoiding coordination by a central server. Due to privacy concerns, DL has become an attractive alternative to centralized learning schemes since training data never leaves the device. In a round of DL, all nodes participate in model training and exchange their model with some other nodes. Performing DL in large-scale heterogeneous networks results in high communication costs and prolonged round durations due to slow nodes, effectively inflating the total training time. Furthermore, current DL algorithms also assume all nodes are available for training and aggregation at all times, diminishing the practicality of DL. This paper presents Plexus, an efficient, scalable, and practical DL system. Plexus (1) avoids network-wide participation by introducing a decentralized peer sampler that selects small subsets of available nodes that train the model each round and, (2) aggregates the trained models produced by nodes every round. Plexus is designed to handle joining and leaving nodes (churn). We extensively evaluate Plexus by incorporating realistic traces for compute speed, pairwise latency, network capacity, and availability of edge devices in our experiments. Our experiments on four common learning tasks empirically show that Plexus reduces time-to-accuracy by 1.2-8.3x, communication volume by 2.4-15.3x and training resources needed for convergence by 6.4-370x compared to baseline DL algorithms.
Semi-decentralized Federated Time Series Prediction with Client Availability Budgets
Department of Electrical and Software Engineering, University of Calgary(电气与软件工程系,卡尔加里大学);Department of Geomatics Engineering, University of Calgary(测绘工程系,卡尔加里大学)
A Multi-Token Coordinate Descent Method for Semi-Decentralized Vertical Federated Learning
Department of Electrical and Computer Engineering, Carnegie Mellon University(卡内基梅隆大学电气与计算机工程系);Institute for Systems and Robotics, Instituto Superior Técnico(系统与机器人研究所,理工学院);Department of Computer Science, NOVA School of Science and Technology(科学与技术学院计算机科学系)
DFPL: Decentralized Federated Prototype Learning Across Heterogeneous Data Distributions
Key Laboratory of Computing Power Network and Information Security, Ministry of Education(教育部算力网络与信息安全重点实验室);Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院));Cyber Security Institute, University of Science and Technology of China(中国科学技术大学网络空间安全学院);Big Data Institute, Qilu University of Technology(齐鲁工业大学大数据研究院)
Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration
Faculty of Applied Sciences, Macao Polytechnic University(澳门理工大学应用科学学院);School of Engineering and Technology, Central Queensland University(中央昆士兰大学工程与技术学院);School of Information and Communication Technology, Griffith University(格里菲斯大学信息与通信技术学院)
S-VOTE: Similarity-based Voting for Client Selection in Decentralized Federated Learning
University of Murcia(穆尔西亚大学);Advantx Technological Foundation (Funditec)(Advantx技术基金会(Funditec));University of Zurich UZH(苏黎世大学);armasuisse Science & Technology(阿马苏斯科学技术局)
DeceFL: A Principled Decentralized Federated Learning Framework
Huazhong University of Science and Technology(华中科技大学);University of Waterloo(滑铁卢大学);Northeastern University(东北大学);KTH Royal Institute of Technology(瑞典皇家理工学院)
Robust Decentralized Personalized Federated Learning via Prediction-Constrained Neighborhood Collaboration
Faculty of Applied Sciences, Macao Polytechnic University(澳门理工大学应用科学学院);School of Engineering and Technology, Central Queensland University(中央昆士兰大学工程与技术学院);School of Information and Communication Technology, Griffith University(格里菲斯大学信息与通信技术学院)
University of Louisville(路易斯维尔大学);North Carolina State University(北卡罗来纳州立大学);University of Wisconsin-Whitewater(威斯康星大学怀特沃特分校);Wayne State University(韦恩州立大学);The Ohio State University(俄亥俄州立大学);IBM Thomas J. Watson Research Center(IBM托马斯·J·沃森研究中心);Duke University(杜克大学)
University of Cambridge(剑桥大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学)
Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling
Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳市人工智能与机器人研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));IBM T. J. Watson Research Center(IBM托马斯·J·沃森研究中心);Yale University(耶鲁大学)
State Key Laboratory of Networking and Switching Technology (BUPT)(网络与交换技术国家重点实验室(北京邮电大学));Key Lab of High Confidence Software Technologies (Peking University)(高可信软件技术重点实验室(北京大学))
Adaptive Heterogeneous Client Sampling for Federated Learning over Wireless Networks
Duke Kunshan University(昆山杜克大学);Carnegie Mellon University(卡内基梅隆大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));IBM T. J. Watson Research Center(IBM T. J. 沃森研究中心);Yale University(耶鲁大学)
SemiDFL: A Semi-Supervised Paradigm for Decentralized Federated Learning
Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人研究院);The Hong Kong Polytechnic University(香港理工大学);Guangdong University of Technology(广东工业大学);Southeast University(东南大学)
Decentralized Directed Collaboration for Personalized Federated Learning
Nanjing University of Science and Technology(南京理工大学);Tsinghua University(清华大学);National University of Defense Technology(国防科技大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));JD Explore Academy(京东探索研究院)