Many recent breakthroughs in deep learning were achieved by training increasingly larger models on massive datasets. However, training such models can be prohibitively expensive. For instance, the cluster used to train GPT-3 costs over \$250 million. As a result, most researchers cannot afford to train state of the art models and contribute to their development. Hypothetically, a researcher could crowdsource the training of large neural networks with thousands of regular PCs provided by volunteers. The raw computing power of a hundred thousand \$2500 desktops dwarfs that of a \$250M server pod, but one cannot utilize that power efficiently with conventional distributed training methods. In this work, we propose Learning@home: a novel neural network training paradigm designed to handle large amounts of poorly connected participants. We analyze the performance, reliability, and architectural constraints of this paradigm and compare it against existing distributed training techniques.
dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysis
School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；PKU-OCTA Laboratory for Blockchain and Privacy Computing（区块链与隐私计算实验室）；National Engineering Research Center for Software Engineering, Peking University（软件工程国家工程研究中心）；State University of New York at Buffalo（纽约州立大学布法罗分校）
In the context of wireless networking, it was recently shown that multiple DNNs can be jointly trained to offer a desired collaborative behaviour capable of coping with a broad range of sensing uncertainties. In particular, it was established that DNNs can be used to derive policies that are robust with respect to the information noise statistic affecting the local information (e.g. CSI in a wireless network) used by each agent (e.g. transmitter) to make its decision. While promising, a major challenge in the implementation of such method is that information noise statistics may differ from ag…
RISE Research Institutes of Sweden（瑞典RISE研究院）；AI Sweden（瑞典人工智能研究院）
In federated learning, clients share a global model that has been trained on decentralized local client data. Although federated learning shows significant promise as a key approach when data cannot be shared or centralized, current methods show limited privacy properties and have shortcomings when applied to common real-world scenarios, especially when client data is heterogeneous. In this paper, we propose an alternative method to learn a personalized model for each client in a federated setting, with greater generalization abilities than previous methods. To achieve this personalization we …
Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The City University of New York（纽约城市大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）
Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approach
Division of Electrical and Computer Engineering, Louisiana State University（电气与计算机工程系，路易斯安那州立大学）；Division of Computer Science and Engineering, Louisiana State University（计算机科学与工程系，路易斯安那州立大学）；Department of Data Science, New Jersey Institute of Technology（数据科学系，新泽西理工学院）；Department of Electrical and Computer Engineering, University of Louisiana at Lafayette（电气与计算机工程系，路易斯安那大学拉斐特分校）；Department of Electrical and Electronic Engineering, Imperial College London（电气与电子工程系，伦敦帝国理工学院）
FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Future Network of Intelligence Institute（未来智联网络研究院）；Guangdong Provincial Key Laboratory of Future Networks of Intelligence（广东省未来智联网络重点实验室）
Multimodal Latent Fusion of ECG Leads for Early Assessment of Pulmonary Hypertension
School of Computer Science, University of Sheffield（谢菲尔德大学计算机科学学院）；Centre for Machine Intelligence, University of Sheffield（谢菲尔德大学智能中心）；Department of Electrical & Computer Science Engineering, IITRAM Ahmedabad（印度阿赫迈德亚布德IITRAM电子与计算机科学工程系）；Digital Environment Research Institute, Queen Mary University of London（伦敦玛丽女王大学数字环境研究院）；Nepal Applied Mathematics and Informatics Institute for research (NAAMII), Nepal（尼泊尔应用数学与信息技术研究所）；Department of Computing, Imperial College London（伦敦帝国学院计算机系）；School of Medicine and Population Health, University of Sheffield（谢菲尔德大学医学与人口健康学院）；Department of Clinical Radiology, Sheffield Teaching Hospitals（谢菲尔德教学医院放射科）；National Institute for Health and Care Research (NIHR), Sheffield Biomedical Research Centre（英国国家健康与护理研究所（NIHR）谢菲尔德生物医学研究中心）
