Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
University of Southern California（南加州大学）
Chaoyang He, Murali Annavaram, Salman Avestimehr
备注：
This paper is accepted to NeurIPS 2020. We propose FedGKT, attempting to address one of the core problems of federated learning: training deep neural networks in resource-constrained edge devices
Scaling up the convolutional neural network (CNN) size (e.g., width, depth, etc.) is known to effectively improve model accuracy. However, the large model size impedes training on resource-constrained edge devices. For instance, federated learning (FL) may place undue burden on the compute capability of edge nodes, even though there is a strong practical need for FL due to its privacy and confidentiality properties. To address the resource-constrained reality of edge devices, we reformulate FL as a group knowledge transfer training algorithm, called FedGKT. FedGKT designs a variant of the alternating minimization approach to train small CNNs on edge nodes and periodically transfer their knowledge by knowledge distillation to a large server-side CNN. FedGKT consolidates several advantages into a single framework: reduced demand for edge computation, lower communication bandwidth for large CNNs, and asynchronous training, all while maintaining model accuracy comparable to FedAvg. We train CNNs designed based on ResNet-56 and ResNet-110 using three distinct datasets (CIFAR-10, CIFAR-100, and CINIC-10) and their non-I.I.D. variants. Our results show that FedGKT can obtain comparable or even slightly higher accuracy than FedAvg. More importantly, FedGKT makes edge training affordable. Compared to the edge training using FedAvg, FedGKT demands 9 to 17 times less computational power (FLOPs) on edge devices and requires 54 to 105 times fewer parameters in the edge CNN. Our source code is released at FedML (https://fedml.ai).
Heterogeneous Federated Learning with Convolutional and Spiking Neural Networks
College of Information Science and Technology, Donghua University（东华大学信息科学与技术学院）；School of Engineering, Westlake University（西湖大学工程学院）；School of Cyber Science and Engineering, Wuhan University（武汉大学网络空间安全学院）
Human activity recognition (HAR) is a machine learning task with important applications in healthcare especially in the context of home care of patients and older adults. HAR is often based on data collected from smart sensors, particularly smart home IoT devices such as smartphones, wearables and other body sensors. Deep learning techniques like convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been used for HAR, both in centralized and federated settings. However, these techniques have certain limitations: RNNs cannot be easily parallelized, CNNs have the limitat…
A Survey of Incremental Transfer Learning: Combining Peer-to-Peer Federated Learning and Domain Incremental Learning for Multicenter Collaboration
University Hospital Erlangen（埃尔朗根大学医院）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Comprehensive Cancer Center Erlangen-EMN（埃尔朗根-EMN综合癌症中心）；Bavarian Cancer Research Center（巴伐利亚癌症研究中心）
Evaluating the Potential of Federated Learning for Maize Leaf Disease Prediction
Institute of Exact and Technological Sciences (IEP)（精确与技术科学研究所）；Federal University of Viçosa (UFV)（维索萨联邦大学）；Faculty of Computing (FACOM)（计算机学院）；Federal University of Uberlândia (UFU)（乌贝兰迪亚联邦大学）
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；IBM Research（IBM研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；University of Michigan（密歇根大学）
Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud. We propose Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures e.g. convolutional neural networks (CNNs) and LSTMs. FedMA constructs the shared global model in a layer-wise manner by matching and averaging hidden elements (i.e. channels for convolution layers; hidden states for LSTM; neurons for fully connected layers) wi…
FedTR: Federated Learning Framework with Transfer Learning for Industrial Visual Inspection
HP-NTU Digital Manufacturing Corporate Lab, Nanyang Technological University（惠普-南洋理工大学数字制造联合实验室，南洋理工大学）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Department of Computer Science, Norwegian University of Science and Technology（挪威科技大学计算机科学系）；HP Inc.（惠普公司）；HP Singapore（惠普新加坡公司）
Exploiting Test-Time Augmentation in Federated Learning for Brain Tumor MRI Classification
Institute of Exact and Technological Sciences, Federal University of Viçosa - UFV, Rio Paranaíba-MG, Brazil（精确与技术科学研究所，弗拉维亚联邦大学-UFV，里奥帕拉纳伊巴-MG，巴西）；Department of Computing, Federal University of São Carlos, São Carlos-SP, Brazil（计算系，萨o卡洛斯联邦大学，萨o卡洛斯-SP，巴西）
Personalized federated learning (FL) aims to train model(s) that can perform well for individual clients that are highly data and system heterogeneous. Most work in personalized FL, however, assumes using the same model architecture at all clients and increases the communication cost by sending/receiving models. This may not be feasible for realistic scenarios of FL. In practice, clients have highly heterogeneous system-capabilities and limited communication resources. In our work, we propose a personalized FL framework, PerFed-CKT, where clients can use heterogeneous model architectures and d…
Federated learning is an approach to train machine learning models on the edge of the networks, as close as possible where the data is produced, motivated by the emerging problem of the inability to stream and centrally store the large amount of data produced by edge devices as well as by data privacy concerns. This learning paradigm is in need of robust algorithms to device heterogeneity and data heterogeneity. This paper proposes ModFL as a federated learning framework that splits the models into a configuration module and an operation module enabling federated learning of the individual mod…
JP Morgan Chase（摩根大通）；University of Cambridge（剑桥大学）
Chest Computational Tomography (CT) scans present low cost, speed and objectivity for COVID-19 diagnosis and deep learning methods have shown great promise in assisting the analysis and interpretation of these images. Most hospitals or countries can train their own models using in-house data, however empirical evidence shows that those models perform poorly when tested on new unseen cases, surfacing the need for coordinated global collaboration. Due to privacy regulations, medical data sharing between hospitals and nations is extremely difficult. We propose a GAN-augmented federated learning m…
In this paper, we investigate the problem of decentralized federated learning (DFL) in Internet of things (IoT) systems, where a number of IoT clients train models collectively for a common task without sharing their private training data in the absence of a central server. Most of the existing DFL schemes are composed of two alternating steps, i.e., model updating and model averaging. However, averaging model parameters directly to fuse different models at the local clients suffers from client-drift especially when the training data are heterogeneous across different clients. This leads to sl…
Deep Transfer Hashing for Adaptive Learning on Federated Streaming Data
Faculty of Computer Science and Business Information Systems, TUAS Würzburg-Schweinfurt（维尔zburg-施魏因富应用科学大学计算机科学与商务信息系统学院）；Faculty of Technology, Bielefeld University（比勒费尔德大学技术学院）；Center for Artificial Intelligence and Robotics Würzburg（维尔zburg人工智能与机器人中心）
Iowa State University（爱荷华州立大学）；IBM Almaden Research Center（IBM阿尔马登研究中心）
Federated learning~(FL) facilitates the training and deploying AI models on edge devices. Preserving user data privacy in FL introduces several challenges, including expensive communication costs, limited resources, and data heterogeneity. In this paper, we propose SPATL, an FL method that addresses these issues by: (a) introducing a salient parameter selection agent and communicating selected parameters only; (b) splitting a model into a shared encoder and a local predictor, and transferring its knowledge to heterogeneous clients via the locally customized predictor. Additionally, we leverage…
Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs
Faculty of Computing, Sabaragamuwa University of Sri Lanka（斯里兰卡萨巴拉加穆瓦大学计算机学院）；Faculty of Engineering, University of Moratuwa（莫拉图瓦大学工程学院）；School of Computing Technologies, RMIT University（RMIT大学计算技术学院）；School of Engineering & Digital Technologies, University of Southern Queensland（南昆士兰大学工程与数字技术学院）；School of Engineering & Technologies, UNSW（新南威尔士大学工程与技术学院）；Faculty of Science and Technology, Charles Darwin University（查尔斯达尔文大学科学技术学院）
Federated Transfer Learning with Differential Privacy
School of Mathematics, University of Birmingham（伯明翰大学数学学院）；Department of Statistics, Columbia University（哥伦比亚大学统计系）；Department of Biostatistics, School of Global Public Health, New York University（纽约大学全球公共卫生学院生物统计系）；Department of Statistics, University of Warwick（华威大学统计系）
Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clients
School of Intelligent Systems Engineering, Sun Yat-Sen University（中山大学智能工程学院）；Shenzhen Fangle Technology Co., Ltd.（深圳方乐科技有限公司）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦玛丽女王大学电子工程与计算机科学学院）
Michigan Technological University（密歇根理工大学）；University of Florida（佛罗里达大学）
Federated learning enables multiple distributed devices to collaboratively learn a shared prediction model without centralizing their on-device data. Most of the current algorithms require comparable individual efforts for local training with the same structure and size of on-device models, which, however, impedes participation from resource-constrained devices. Given the widespread yet heterogeneous devices nowadays, in this paper, we propose an innovative federated learning framework with heterogeneous on-device models through Zero-shot Knowledge Transfer, named by FedZKT. Specifically, FedZ…
Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）
Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to be deployed across the clients and server, making it infeasible to train large models due to clients' limited system resources. In this work, we propose a novel ensemble knowledge transfer method named Fed-ET in which small models (different in architecture) are trained on clients, and used to train a larger model at the server. Unlike in conventional ensemble learning, in FL …
Federated learning is a decentralized approach for training models on distributed devices, by summarizing local changes and sending aggregate parameters from local models to the cloud rather than the data itself. In this research we employ the idea of transfer learning to federated training for next word prediction (NWP) and conduct a number of experiments demonstrating enhancements to current baselines for which federated NWP models have been successful. Specifically, we compare federated training baselines from randomly initialized models to various combinations of pretraining approaches inc…
University of Science and Technology of China (USTC)（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）
Attributes skew hinders the current federated learning (FL) frameworks from consistent optimization directions among the clients, which inevitably leads to performance reduction and unstable convergence. The core problems lie in that: 1) Domain-specific attributes, which are non-causal and only locally valid, are indeliberately mixed into global aggregation. 2) The one-stage optimizations of entangled attributes cannot simultaneously satisfy two conflicting objectives, i.e., generalization and personalization. To cope with these, we proposed disentangled federated learning (DFL) to disentangle…
Arizona State University（亚利桑那州立大学）；Shanghai Jiao Tong University（上海交通大学）
This work aims to tackle Model Inversion (MI) attack on Split Federated Learning (SFL). SFL is a recent distributed training scheme where multiple clients send intermediate activations (i.e., feature map), instead of raw data, to a central server. While such a scheme helps reduce the computational load at the client end, it opens itself to reconstruction of raw data from intermediate activation by the server. Existing works on protecting SFL only consider inference and do not handle attacks during training. So we propose ResSFL, a Split Federated Learning Framework that is designed to be MI-re…
CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning
Kyung Hee University（庆熙大学）；The University of Danang—Vietnam-Korea University of Information and Communication Technology（岘港大学—越南-韩国信息与通信技术大学）；Aalborg University（奥尔堡大学）
Adaptive Guidance for Local Training in Heterogeneous Federated Learning
Shanghai Jiao Tong University（上海交通大学）；Hong Kong Polytechnic University（香港理工大学）；Queen's University Belfast（贝尔法斯特女王大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Key Laboratory of Trusted Data Circulation（上海可信数据流通关键实验室）
For intelligent home IoT services with sensors and machine learning, we need to upload IoT data to the cloud server which cannot share private data for training. A recent machine learning approach, called federated learning, keeps user data on the device in the distributed computing environment. Though federated learning is useful for protecting privacy, it experiences poor performance in terms of the end-to-end response time in home IoT services, because IoT devices are usually controlled by remote servers in the cloud. In addition, it is difficult to achieve the high accuracy of federated le…
FedCL-Ensemble Learning: A Framework of Federated Continual Learning with Ensemble Transfer Learning Enhanced for Alzheimer's MRI Classifications while Preserving Privacy
The Hong Kong Polytechnic University（香港理工大学）；HUST（华中科技大学）
In recent years, personalized federated learning (pFL) has attracted increasing attention for its potential in dealing with statistical heterogeneity among clients. However, the state-of-the-art pFL methods rely on model parameters aggregation at the server side, which require all models to have the same structure and size, and thus limits the application for more heterogeneous scenarios. To deal with such model constraints, we exploit the potentials of heterogeneous model settings and propose a novel training framework to employ personalized models for different clients. Specifically, we form…
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Academy of Sciences（中国科学院）；Institute of Electronics, Chinese Academy of Sciences（中国科学院电子学研究所）
Deep convolutional neural networks (DCNNs) have attracted much attention in remote sensing recently. Compared with the large-scale annotated dataset in natural images, the lack of labeled data in remote sensing becomes an obstacle to train a deep network very well, especially in SAR image interpretation. Transfer learning provides an effective way to solve this problem by borrowing the knowledge from the source task to the target task. In optical remote sensing application, a prevalent mechanism is to fine-tune on an existing model pre-trained with a large-scale natural image dataset, such as …
Global AI Accelerator, Ericsson（爱立信全球人工智能加速器）；Ericsson Research（爱立信研究院）
Various health-care applications such as assisted living, fall detection, etc., require modeling of user behavior through Human Activity Recognition (HAR). Such applications demand characterization of insights from multiple resource-constrained user devices using machine learning techniques for effective personalized activity monitoring. On-device Federated Learning proves to be an effective approach for distributed and collaborative machine learning. However, there are a variety of challenges in addressing statistical (non-IID data) and model heterogeneities across users. In addition, in this…
University Of Texas at Arlington（德克萨斯大学阿灵顿分校）；Kuaishou Techlology（快手科技）
Unsupervised domain adaptation (UDA) aims to transfer the knowledge learnt from a labeled source domain to an unlabeled target domain. Previous work is mainly built upon convolutional neural networks (CNNs) to learn domain-invariant representations. With the recent exponential increase in applying Vision Transformer (ViT) to vision tasks, the capability of ViT in adapting cross-domain knowledge, however, remains unexplored in the literature. To fill this gap, this paper first comprehensively investigates the transferability of ViT on a variety of domain adaptation tasks. Surprisingly, ViT demo…
The Federated Learning (FL) workflow of training a centralized model with distributed data is growing in popularity. However, until recently, this was the realm of contributing clients with similar computing capability. The fast expanding IoT space and data being generated and processed at the edge are encouraging more effort into expanding federated learning to include heterogeneous systems. Previous approaches distribute light-weight models to clients are rely on knowledge transfer to distil the characteristic of local data in partitioned updates. However, their additional knowledge exchange…
Federated learning is a model for privacy without revealing private data by transfer models instead of personal and private data from local client devices. While, in the global model, it's crucial to recognize each local data is normal. This paper suggests one method to separate normal locals and abnormal locals by Euclidean similarity clustering of vectors extracted by inputting dummy data in local models. In a federated classification model, this method divided locals into normal and abnormal.
École de Technologie Supérieure (ÉTS)（高等技术学院（ÉTS））；Universite de Quebec a Trois-Rivieres (UQTR)（魁北克大学三河分校（UQTR））；Lebanese American University（黎巴嫩美国大学）；New York University（纽约大学）；Mohammad Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Continuous behavioural authentication methods add a unique layer of security by allowing individuals to verify their unique identity when accessing a device. Maintaining session authenticity is now feasible by monitoring users' behaviour while interacting with a mobile or Internet of Things (IoT) device, making credential theft and session hijacking ineffective. Such a technique is made possible by integrating the power of artificial intelligence and Machine Learning (ML). Most of the literature focuses on training machine learning for the user by transmitting their data to an external server,…
Automatically mining sentiment tendency contained in natural language is a fundamental research to some artificial intelligent applications, where solutions alternate with challenges. Transfer learning and multi-task learning techniques have been leveraged to mitigate the supervision sparsity and collaborate multiple heterogeneous domains correspondingly. Recent years, the sensitive nature of users' private data raises another challenge for sentiment classification, i.e., data privacy protection. In this paper, we resort to federated learning for multiple domain sentiment classification under …
We show that a word-level recurrent neural network can predict emoji from text typed on a mobile keyboard. We demonstrate the usefulness of transfer learning for predicting emoji by pretraining the model using a language modeling task. We also propose mechanisms to trigger emoji and tune the diversity of candidates. The model is trained using a distributed on-device learning framework called federated learning. The federated model is shown to achieve better performance than a server-trained model. This work demonstrates the feasibility of using federated learning to train production-quality mo…
