Scaling Neuroscience Research using Federated Learning
Information Sciences Institute（信息科学研究所）；University of Southern California（南加州大学）；Imaging Genetics Center（影像遗传学中心）；USC Stevens Neuroimaging and Informatics Institute（南加州大学史蒂文斯神经影像与信息学研究所）
Dimitris Stripelis, Jose Luis Ambite, Pradeep Lam, Paul Thompson
备注：
To appear at IEEE International Symposium on Biomedical Imaging 2021 (ISBI 2021)
The amount of biomedical data continues to grow rapidly. However, the ability to analyze these data is limited due to privacy and regulatory concerns. Machine learning approaches that require data to be copied to a single location are hampered by the challenges of data sharing. Federated Learning is a promising approach to learn a joint model over data silos. This architecture does not share any subject data across sites, only aggregated parameters, often in encrypted environments, thus satisfying privacy and regulatory requirements. Here, we describe our Federated Learning architecture and training policies. We demonstrate our approach on a brain age prediction model on structural MRI scans distributed across multiple sites with diverse amounts of data and subject (age) distributions. In these heterogeneous environments, our Semi-Synchronous protocol provides faster convergence.
Peltarion（佩尔塔里昂）；RISE Research Institutes of Sweden（瑞典RISE研究院）
Federated learning (FL) is a promising approach to distributed compute, as well as distributed data, and provides a level of privacy and compliance to legal frameworks. This makes FL attractive for both consumer and healthcare applications. While the area is actively being explored, few studies have examined FL in the context of larger language models and there is a lack of comprehensive reviews of robustness across tasks, architectures, numbers of clients, and other relevant factors. In this paper, we explore the fine-tuning of Transformer-based language models in a federated learning setting…
Improving Local Training in Federated Learning via Temperature Scaling
School of Integrated Technology, Yonsei University（燕尼大学整合技术学院）；BK21 Graduate Program in Intelligent Semiconductor Technology, Yonsei University（燕尼大学智能半导体技术研究生项目）；Department of Electrical Engineering and Computer Science, University of Michigan（密歇根大学电气工程与计算机科学系）
Most studies in cross-device federated learning focus on small models, due to the server-client communication and on-device computation bottlenecks. In this work, we leverage various techniques for mitigating these bottlenecks to train larger language models in cross-device federated learning. With systematic applications of partial model training, quantization, efficient transfer learning, and communication-efficient optimizers, we are able to train a $21$M parameter Transformer and $20.2$M parameter Conformer that achieve the same or better perplexity as that of a similarly sized LSTM with $…
Deakin University（迪肯大学）；Commonwealth Scientific and Industrial Research Organization（澳大利亚联邦科学与工业研究组织）；Xidian University（西安电子科技大学）；Qilu University of Technology（齐鲁工业大学）；Shandong Computer Science Center（山东省计算中心）
Since the traffic conditions change over time, machine learning models that predict traffic flows must be updated continuously and efficiently in smart public transportation. Federated learning (FL) is a distributed machine learning scheme that allows buses to receive model updates without waiting for model training on the cloud. However, FL is vulnerable to poisoning or DDoS attacks since buses travel in public. Some work introduces blockchain to improve reliability, but the additional latency from the consensus process reduces the efficiency of FL. Asynchronous Federated Learning (AFL) is a …
National Institute of Science, Education and Research（国家科学教育与研究学院）；University of Kalyani（卡利亚尼大学）；Homi Bhabha National Institute（霍米·巴巴国家研究院）
Federated learning (FL) is a distributed learning process where the model (weights and checkpoints) is transferred to the devices that posses data rather than the classical way of transferring and aggregating the data centrally. In this way, sensitive data does not leave the user devices. FL uses the FedAvg algorithm, which is trained in the iterative model averaging way, on the non-iid and unbalanced distributed data, without depending on the data quantity. Some issues with the FL are, 1) no scalability, as the model is iteratively trained over all the devices, which amplifies with device dro…
Rethinking Federated Learning Over the Air: The Blessing of Scaling Up
Zhejiang University/University of Illinois Urbana-Champaign Institute（浙江大学/伊利诺伊大学厄巴纳-香槟分校研究所）；Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Engineering Systems and Design, Singapore University of Technology and Design（新加坡科技与设计大学工程系统与设计系）；School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机科学学院）；Department of Computer and Information Science, Linköping University（_linköping大学计算机与信息科学系）
FAROS: Robust Federated Learning with Adaptive Scaling against Backdoor Attacks
Southwest University（西南大学）；University of Electronic Science and Technology of China（电子科技大学）；Kobe University（Kobe大学）；Zhongguancun Laboratory（中关村实验室）；Waseda University（早稻田大学）
This paper considers over-the-air federated learning (OTA-FL). OTA-FL exploits the superposition property of the wireless medium, and performs model aggregation over the air for free. Thus, it can greatly reduce the communication cost incurred in communicating model updates from the edge devices. In order to fully utilize this advantage while providing comparable learning performance to conventional federated learning that presumes model aggregation via noiseless channels, we consider the joint design of transmission scaling and the number of local iterations at each round, given the power con…
School of Mathematical Sciences, Peking University（北京大学数学科学学院）
Federated Learning (FL) makes a large amount of edge computing devices (e.g., mobile phones) jointly learn a global model without data sharing. In FL, data are generated in a decentralized manner with high heterogeneity. This paper studies how to perform statistical estimation and inference in the federated setting. We analyze the so-called Local SGD, a multi-round estimation procedure that uses intermittent communication to improve communication efficiency. We first establish a {\it functional central limit theorem} that shows the averaged iterates of Local SGD weakly converge to a rescaled B…
Cross-device Federated Learning is an increasingly popular machine learning setting to train a model by leveraging a large population of client devices with high privacy and security guarantees. However, communication efficiency remains a major bottleneck when scaling federated learning to production environments, particularly due to bandwidth constraints during uplink communication. In this paper, we formalize and address the problem of compressing client-to-server model updates under the Secure Aggregation primitive, a core component of Federated Learning pipelines that allows the server to …
Cross-device Federated Learning (FL) is a distributed learning paradigm with several challenges that differentiate it from traditional distributed learning, variability in the system characteristics on each device, and millions of clients coordinating with a central server being primary ones. Most FL systems described in the literature are synchronous - they perform a synchronized aggregation of model updates from individual clients. Scaling synchronous FL is challenging since increasing the number of clients training in parallel leads to diminishing returns in training speed, analogous to lar…
Microsoft Research（微软研究院）；University of California, San Diego（加州大学圣地亚哥分校）
In this paper we introduce "Federated Learning Utilities and Tools for Experimentation" (FLUTE), a high-performance open-source platform for federated learning research and offline simulations. The goal of FLUTE is to enable rapid prototyping and simulation of new federated learning algorithms at scale, including novel optimization, privacy, and communications strategies. We describe the architecture of FLUTE, enabling arbitrary federated modeling schemes to be realized. We compare the platform with other state-of-the-art platforms and describe available features of FLUTE for experimentation i…
Scaling Probabilistic Circuits via Data Partitioning
TU Darmstadt（达姆施塔特工业大学）；Hessian Center for AI (hessian.AI)（黑森人工智能中心）；Eindhoven University of Technology（埃因霍温理工大学）；University of Bremen（不来梅大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）
Carnegie Mellon University（卡内基梅隆大学）；Massachusetts Institute of Technology（麻省理工学院）；Columbia University（哥伦比亚大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Petuum Inc.（Petuum公司）
The canonical formulation of federated learning treats it as a distributed optimization problem where the model parameters are optimized against a global loss function that decomposes across client loss functions. A recent alternative formulation instead treats federated learning as a distributed inference problem, where the goal is to infer a global posterior from partitioned client data (Al-Shedivat et al., 2021). This paper extends the inference view and describes a variational inference formulation of federated learning where the goal is to find a global variational posterior that well-app…
Distributed training across several quantum computers could significantly improve the training time and if we could share the learned model, not the data, it could potentially improve the data privacy as the training would happen where the data is located. However, to the best of our knowledge, no work has been done in quantum machine learning (QML) in federation setting yet. In this work, we present the federated training on hybrid quantum-classical machine learning models although our framework could be generalized to pure quantum machine learning model. Specifically, we consider the quantum…
We tackle the problem of Federated Learning in the non i.i.d. case, in which local models drift apart, inhibiting learning. Building on an analogy with Lifelong Learning, we adapt a solution for catastrophic forgetting to Federated Learning. We add a penalty term to the loss function, compelling all local models to converge to a shared optimum. We show that this can be done efficiently for communication (adding no further privacy risks), scaling with the number of nodes in the distributed setting. Our experiments show that this method is superior to competing ones for image recognition on the …
Federated learning (FL) is a new distributed machine learning framework that can achieve reliably collaborative training without collecting users' private data. However, due to FL's frequent communication and average aggregation strategy, they experience challenges scaling to statistical diversity data and large-scale models. In this paper, we propose a personalized FL framework, named Tensor Decomposition based Personalized Federated learning (TDPFed), in which we design a novel tensorized local model with tensorized linear layers and convolutional layers to reduce the communication cost. TDP…
Federated learning (FL) is an attractive paradigm for making use of rich distributed data while protecting data privacy. Nonetheless, nonideal communication links and limited transmission resources may hinder the implementation of fast and accurate FL. In this paper, we study joint optimization of communications and FL based on analog aggregation transmission in realistic wireless networks. We first derive closed-form expressions for the expected convergence rate of FL over the air, which theoretically quantify the impact of analog aggregation on FL. Based on the analytical results, we develop…
Inner Mongolia University of Technology（内蒙古工业大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；New Jersey Institute of Technology（新泽西理工学院）
Advanced researches on connected vehicles have recently targeted to the integration of vehicle-to-everything (V2X) networks with Machine Learning (ML) tools and distributed decision making. Federated learning (FL) is emerging as a new paradigm to train machine learning (ML) models in distributed systems, including vehicles in V2X networks. Rather than sharing and uploading the training data to the server, the updating of model parameters (e.g., neural networks' weights and biases) is applied by large populations of interconnected vehicles, acting as local learners. Despite these benefits, the …
FedOBD: Opportunistic Block Dropout for Efficiently Training Large-scale Neural Networks through Federated Learning
School of Computer Science and Engineering, Nanyang Technological University, Singapore（南洋理工大学计算机科学与工程学院）；University of California, Santa Barbara, CA, USA（加州大学圣巴巴拉分校）
FUSE: Fairness-aware Update Screening against Byzantine Exploitation in $q$-Fair Federated Learning
George Mason University（乔治梅森大学）；The University of Hong Kong（香港大学）；Huazhong University of Science and Technology（华中科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The Hong Kong Polytechnic University（香港理工大学）
FedPOD: the deployable units of training for federated learning
Interdisciplinary Program in Bioengineering, College of Engineering, Seoul National University, Seoul, South Korea（生物工程跨学科项目、工程学院、首尔国立大学、首尔、韩国）；Department of Nuclear Medicine, College of Medicine, Seoul National University, Seoul, South Korea（核医学系、医学院、首尔国立大学、首尔、韩国）
FCNCP: A Coupled Nonnegative CANDECOMP/PARAFAC Decomposition Based on Federated Learning
School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology（大连理工大学医学院生物医学工程学院）；Key Laboratory of Integrated Circuit and Biomedical Electronic System, Liaoning Province, Dalian University of Technology（大连理工大学辽宁省集成电路与生物医学电子系统重点实验室）；Stem Cell Clinical Research Center, the First Affiliated Hospital of Dalian Medical University（大连医科大学附属第一医院干细胞临床研究中心）；Dalian Innovation Institute of Stem Cell and Precision Medicine（大连干细胞与精准医学创新研究院）；Key Laboratory of Social Computing and Cognitive Intelligence (Dalian University of Technology), Ministry of Education, China（教育部社会计算与认知智能重点实验室（大连理工大学））
The University of British Columbia（不列颠哥伦比亚大学）；Microsoft Research（微软研究院）
The widespread adoption of handheld devices have fueled rapid growth in new applications. Several of these new applications employ machine learning models to train on user data that is typically private and sensitive. Federated Learning enables machine learning models to train locally on each handheld device while only synchronizing their neuron updates with a server. While this enables user privacy, technology scaling and software advancements have resulted in handheld devices with varying performance capabilities. This results in the training time of federated learning tasks to be dictated b…
Federated learning (FL) enables distributed computation of machine learning models over various disparate, remote data sources, without requiring to transfer any individual data to a centralized location. This results in an improved generalizability of models and efficient scaling of computation as more sources and larger datasets are added to the federation. Nevertheless, recent membership attacks show that private or sensitive personal data can sometimes be leaked or inferred when model parameters or summary statistics are shared with a central site, requiring improved security solutions. In…
