Over-the-Air (OtA) Federated Learning (FL) refers to an FL system where multiple agents apply OtA computation for transmitting model updates to a common edge server. Two important features of OtA computation, namely linear processing and signal-level superposition, motivate the use of linear compression with compressed sensing (CS) methods to reduce the number of data samples transmitted over the channel. The previous works on applying CS methods in OtA FL have primarily assumed that the original model update vectors are sparse, or they have been sparsified before compression. However, it is unclear whether linear compression with CS-based reconstruction is more effective than directly sending the non-zero elements in the sparsified update vectors, under the same total power constraint. In this study, we examine and compare several communication designs with or without sparsification. Our findings demonstrate that sparsification before compression is not necessary. Alternatively, sparsification without linear compression can also achieve better performance than the commonly considered setup that combines both.
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；Imperial College London（伦敦帝国理工学院）
Federated edge learning (FEEL) is a promising distributed machine learning (ML) framework to drive edge intelligence applications. However, due to the dynamic wireless environments and the resource limitations of edge devices, communication becomes a major bottleneck. In this work, we propose time-correlated sparsification with hybrid aggregation (TCS-H) for communication-efficient FEEL, which exploits jointly the power of model compression and over-the-air computation. By exploiting the temporal correlations among model parameters, we construct a global sparsification mask, which is identical…
Imperial College London（帝国理工学院）；Ozyegin University（奥兹耶金大学）
Federated learning (FL) enables multiple clients to collaboratively train a shared model without disclosing their local datasets. This is achieved by exchanging local model updates with the help of a parameter server (PS). However, due to the increasing size of the trained models, the communication load due to the iterative exchanges between the clients and the PS often becomes a bottleneck in the performance. Sparse communication is often employed to reduce the communication load, where only a small subset of the model updates are communicated from the clients to the PS. In this paper, we int…
Princeton University（普林斯顿大学）；University of Chicago（芝加哥大学）；IBM
Federated learning is inherently vulnerable to model poisoning attacks because its decentralized nature allows attackers to participate with compromised devices. In model poisoning attacks, the attacker reduces the model's performance on targeted sub-tasks (e.g. classifying planes as birds) by uploading "poisoned" updates. In this report we introduce \algoname{}, a novel defense that uses global top-k update sparsification and device-level gradient clipping to mitigate model poisoning attacks. We propose a theoretical framework for analyzing the robustness of defenses against poisoning attacks…
Mobility-Aware Asynchronous Federated Learning with Dynamic Sparsification
Beijing National Research Center for Information Science and Technology, Department of Electronic Engineering, Tsinghua University（北京信息科学与技术国家研究中心，电子工程系，清华大学）；School of Electronic and Information Engineering, Beijing Jiaotong University（电子与信息工程学院，北京交通大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
User-level differential privacy (DP) provides certifiable privacy guarantees to the information that is specific to any user's data in federated learning. Existing methods that ensure user-level DP come at the cost of severe accuracy decrease. In this paper, we study the cause of model performance degradation in federated learning under user-level DP guarantee. We find the key to solving this issue is to naturally restrict the norm of local updates before executing operations that guarantee DP. To this end, we propose two techniques, Bounded Local Update Regularization and Local Update Sparsif…
The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Federated learning (FL) enables distributed agents to collaboratively learn a centralized model without sharing their raw data with each other. However, data locality does not provide sufficient privacy protection, and it is desirable to facilitate FL with rigorous differential privacy (DP) guarantee. Existing DP mechanisms would introduce random noise with magnitude proportional to the model size, which can be quite large in deep neural networks. In this paper, we propose a new FL framework with sparsification-amplified privacy. Our approach integrates random sparsification with gradient pert…
Federated Cubic Regularized Newton Learning with Sparsification-amplified Differential Privacy
Hong Kong University of Science and Technology（香港科技大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）；Southern University of Science and Technology（南方科技大学）
We explore a scheme that enables the training of a deep neural network in a Federated Learning configuration over an additive white Gaussian noise channel. The goal is to create a low complexity, linear compression strategy, called PolarAir, that reduces the size of the gradient at the user side to lower the number of channel uses needed to transmit it. The suggested approach belongs to the family of compressed sensing techniques, yet it constructs the sensing matrix and the recovery procedure using multiple access techniques. Simulations show that it can reduce the number of channel uses by ~…
Privacy-preserving Federated Primal-dual Learning for Non-convex and Non-smooth Problems with Model Sparsification
Institute of Communications Engineering, National Tsing Hua University（国立清华大学通信工程研究所）；University of Electronic Science and Technology of China（电子科技大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Yonsei University（延世大学）
China University of Political Science Law（中国政法大学）；Xi’an Jiaotong University（西安交通大学）；Ant Rongxin (Chengdu) Network Technology Co Ltd（蚂蚁融信（成都）网络科技有限公司）
The conventional machine learning (ML) and deep learning approaches need to share customers' sensitive information with an external credit bureau to generate a prediction model that opens the door to privacy leakage. This leakage risk makes financial companies face an enormous challenge in their cooperation. Federated learning is a machine learning setting that can protect data privacy, but the high communication cost is often the bottleneck of the federated systems, especially for large neural networks. Limiting the number and size of communications is necessary for the practical training of …
Sparsification Under Siege: Dual-Level Defense Against Poisoning in Communication-Efficient Federated Learning
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Cyber Science and Technology at Beihang University（北京航空航天大学网络安全科学与技术学院）；School of Cyberspace Science and Technology, Beijing Jiaotong University（北京交通大学网络空间科学与技术学院）；Beijing Key Laboratory of Security and Privacy in Intelligent Transportation（北京智能交通安全与隐私重点实验室）；Zhongguancun Laboratory（中关村实验室）；School of Computing and Information,University of Pittsburgh（匹兹堡大学计算与信息学院）
Imperial College London（帝国理工学院）；IBM T. J. Watson Research Center（IBM T·J·沃森研究中心）
Federated learning (FL) is an emerging technique for training machine learning models using geographically dispersed data collected by local entities. It includes local computation and synchronization steps. To reduce the communication overhead and improve the overall efficiency of FL, gradient sparsification (GS) can be applied, where instead of the full gradient, only a small subset of important elements of the gradient is communicated. Existing work on GS uses a fixed degree of gradient sparsity for i.i.d.-distributed data within a datacenter. In this paper, we consider adaptive degree of s…
Efficient Asynchronous Federated Learning with Sparsification and Quantization
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Hithink RoyalFlush Information Network Co., Ltd.（同花顺信息服务股份有限公司）；Collaborative Innovation Center of Novel Software Technology and Industrialization（新型软件技术及产业化协同创新中心）；Department of Sciences and Informatics, Muroran Institute of Technology（室兰工业大学科学与信息学系）；Boston Consulting Group（波士顿咨询公司）
We use gradient sparsification to reduce the adverse effect of differential privacy noise on performance of private machine learning models. To this aim, we employ compressed sensing and additive Laplace noise to evaluate differentially-private gradients. Noisy privacy-preserving gradients are used to perform stochastic gradient descent for training machine learning models. Sparsification, achieved by setting the smallest gradient entries to zero, can reduce the convergence speed of the training algorithm. However, by sparsification and compressed sensing, the dimension of communicated gradien…
Secure aggregation is a popular protocol in privacy-preserving federated learning, which allows model aggregation without revealing the individual models in the clear. On the other hand, conventional secure aggregation protocols incur a significant communication overhead, which can become a major bottleneck in real-world bandwidth-limited applications. Towards addressing this challenge, in this work we propose a lightweight gradient sparsification framework for secure aggregation, in which the server learns the aggregate of the sparsified local model updates from a large number of users, but w…
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）
In the recent paper FLECS (Agafonov et al, FLECS: A Federated Learning Second-Order Framework via Compression and Sketching), the second-order framework FLECS was proposed for the Federated Learning problem. This method utilize compression of sketched Hessians to make communication costs low. However, the main bottleneck of FLECS is gradient communication without compression. In this paper, we propose the modification of FLECS with compressed gradient differences, which we call FLECS-CGD (FLECS with Compressed Gradient Differences) and make it applicable for stochastic optimization. Convergenc…
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；School of Computing, Faculty of Science and Engineering, Macquarie University（麦考瑞大学科学与工程学院计算机学院）；Peng Cheng Laboratory（鹏城实验室）；School of Computing Science, Simon Fraser University（西蒙弗雷泽大学计算科学学院）
Federated Learning (FL) incurs high communication overhead, which can be greatly alleviated by compression for model updates. Yet the tradeoff between compression and model accuracy in the networked environment remains unclear and, for simplicity, most implementations adopt a fixed compression rate only. In this paper, we for the first time systematically examine this tradeoff, identifying the influence of the compression error on the final model accuracy with respect to the learning rate. Specifically, we factor the compression error of each global iteration into the convergence rate analysis…
Homomorphic encryption (HE) is a promising privacy-preserving technique for cross-silo federated learning (FL), where organizations perform collaborative model training on decentralized data. Despite the strong privacy guarantee, general HE schemes result in significant computation and communication overhead. Prior works employ batch encryption to address this problem, but it is still suboptimal in mitigating communication overhead and is incompatible with sparsification techniques. In this paper, we propose FLASHE, an HE scheme tailored for cross-silo FL. To capture the minimum requirements o…
Imperial College London（帝国理工学院）；Sabanci University（萨班哲大学）
We study collaborative machine learning (ML) across wireless devices, each with its own local dataset. Offloading these datasets to a cloud or an edge server to implement powerful ML solutions is often not feasible due to latency, bandwidth and privacy constraints. Instead, we consider federated edge learning (FEEL), where the devices share local updates on the model parameters rather than their datasets. We consider a heterogeneous cellular network (HCN), where small cell base stations (SBSs) orchestrate FL among the mobile users (MUs) within their cells, and periodically exchange model updat…
We provide the optimal receive combining strategy for federated learning in multiple-input multiple-output (MIMO) systems. Our proposed algorithm allows the clients to perform individual gradient sparsification which greatly improves performance in scenarios with heterogeneous (non i.i.d.) training data. The proposed method beats the benchmark by a wide margin.
Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization
Pillar of Information Systems Technology and Design（信息系统技术与设计学院）；School of Information Science and Engineering（信息科学与工程学院）；School of Cyber Science and Engineering（网络安全科学与工程学院）；National Mobile Communications Research Laboratory（国家移动通信研究实验室）；School of Electronic and Optical Engineering（电子与光学工程学院）；School of Electronics, Electrical Engineering and Computer Science (EEECS)（电子、电气工程与计算机科学学院）；Data61, CSIRO（Data61，CSIRO）；Department of Electronics Engineering（电子工程系）；Department of Electrical and Computer Engineering（电气与计算机工程系）
Rice University（莱斯大学）；University of Houston-Downtown（休斯顿大学市中心分校）；Prudence Medical Technologies Ltd.（审慎医疗技术有限公司）
The Internet of Things (IoT) revolution has shown potential to give rise to many medical applications with access to large volumes of healthcare data collected by IoT devices. However, the increasing demand for healthcare data privacy and security makes each IoT device an isolated island of data. Further, the limited computation and communication capacity of wearable healthcare devices restrict the application of vanilla federated learning. To this end, we propose an advanced federated learning framework to train deep neural networks, where the network is partitioned and allocated to IoT devic…
Modern large scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information, such as stochastic gradients, among different nodes. Recently, gradient sparsification techniques have been proposed to reduce communications cost and thus alleviate the network overhead. However, most of gradient sparsification techniques consider only synchronous parallelism and cannot be applied in asynchronous scenarios, such as asynchronous distributed training fo…
Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face are heterogeneity in data across clients and collaboration of clients with {\em diverse resources}. In this work, we introduce a \textit{quantized} and \textit{personalized} FL algorithm QuPeD that facilitates collective (personalized model compression) training via \textit{knowledge distillation} (KD) among clients who have access to heterogeneous data and resources. For personalization, we allow clients to learn \…
Model compression is important in federated learning (FL) with large models to reduce communication cost. Prior works have been focusing on sparsification based compression that could desparately affect the global model accuracy. In this work, we propose a new scheme for upstream communication where instead of transmitting the model update, each client learns and transmits a light-weight synthetic dataset such that using it as the training data, the model performs similarly well on the real training data. The server will recover the local model update via the synthetic data and apply standard …
Federated learning can enable remote workers to collaboratively train a shared machine learning model while allowing training data to be kept locally. In the use case of wireless mobile devices, the communication overhead is a critical bottleneck due to limited power and bandwidth. Prior work has utilized various data compression tools such as quantization and sparsification to reduce the overhead. In this paper, we propose a predictive coding based compression scheme for federated learning. The scheme has shared prediction functions among all devices and allows each worker to transmit a compr…
Northeastern University（东北大学）；Duke University（杜克大学）；University of Connecticut（康涅狄格大学）
Nowadays, Deep Neural Networks are widely applied to various domains. However, massive data collection required for deep neural network reveals the potential privacy issues and also consumes large mounts of communication bandwidth. To address these problems, we propose a privacy-preserving method for the federated learning distributed system, operated on Intel Software Guard Extensions, a set of instructions that increase the security of application code and data. Meanwhile, the encrypted models make the transmission overhead larger. Hence, we reduce the commutation cost by sparsification and …
Zhejiang University（浙江大学）；Wuhan University（武汉大学）；Singapore University of Technology and Design（新加坡科技设计大学）；North Carolina State University（北卡罗来纳州立大学）
Communication overhead has become one of the major bottlenecks in the distributed training of deep neural networks. To alleviate the concern, various gradient compression methods have been proposed, and sign-based algorithms are of surging interest. However, SIGNSGD fails to converge in the presence of data heterogeneity, which is commonly observed in the emerging federated learning (FL) paradigm. Error feedback has been proposed to address the non-convergence issue. Nonetheless, it requires the workers to locally keep track of the compression errors, which renders it not suitable for FL since…
We investigate the problem of private read update write (PRUW) in federated submodel learning (FSL) with sparsification. In FSL, a machine learning model is divided into multiple submodels, where each user updates only the submodel that is relevant to the user's local data. PRUW is the process of privately performing FSL by reading from and writing to the required submodel without revealing the submodel index, or the values of updates to the databases. Sparsification is a widely used concept in learning, where the users update only a small fraction of parameters to reduce the communication cos…
We consider a standard federated learning (FL) architecture where a group of clients periodically coordinate with a central server to train a statistical model. We develop a general algorithmic framework called FedLin to tackle some of the key challenges intrinsic to FL, namely objective heterogeneity, systems heterogeneity, and infrequent and imprecise communication. Our framework is motivated by the observation that under these challenges, various existing FL algorithms suffer from a fundamental speed-accuracy conflict: they either guarantee linear convergence but to an incorrect point, or c…
Compressed Proximal Federated Learning for Non-Convex Composite Optimization on Heterogeneous Data
School of Intelligent Systems Engineering, Sun Yat-Sen University（中山大学智能系统工程学院）；Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学自动化系）；Department of Mechanical Engineering, University of Victoria（维多利亚大学机械工程系）
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM Thomas J. Watson Research Center（IBM托马斯·J·沃森研究中心）
We propose Compressed Vertical Federated Learning (C-VFL) for communication-efficient training on vertically partitioned data. In C-VFL, a server and multiple parties collaboratively train a model on their respective features utilizing several local iterations and sharing compressed intermediate results periodically. Our work provides the first theoretical analysis of the effect message compression has on distributed training over vertically partitioned data. We prove convergence of non-convex objectives at a rate of $O(\frac{1}{\sqrt{T}})$ when the compression error is bounded over the course…
Communication-efficient Vertical Federated Learning via Compressed Error Feedback
Carnegie Mellon University（卡内基梅隆大学）；Institute for Systems and Robotics（系统与机器人研究所）；Instituto Superior Técnico（里斯本高等理工学院）；NOVA School of Science and Technology（NOVA科学技术学院）
Sejong University（世宗大学）；Pusan National University（釜山国立大学）
Recently, a considerable amount of works have been made to tackle the communication burden in federated learning (FL) (e.g., model quantization, data sparsification, and model compression). However, the existing methods, that boost the communication efficiency in FL, result in a considerable trade-off between communication efficiency and global convergence rate. We formulate an optimization problem for compression-aided FL, which captures the relationship between the distortion rate, number of participating IoT devices, and convergence rate. Following that, the objective function is to minimiz…
Fraunhofer IVI（弗劳恩霍夫交通与基础设施系统研究所）；Technical University of Munich（慕尼黑工业大学）；Sony AI（索尼人工智能研究院）；Technische Hochschule Ingolstadt（英戈尔施塔特应用技术大学）
Federated learning enables cooperative training among massively distributed clients by sharing their learned local model parameters. However, with increasing model size, deploying federated learning requires a large communication bandwidth, which limits its deployment in wireless networks. To address this bottleneck, we introduce a residual-based federated learning framework (ResFed), where residuals rather than model parameters are transmitted in communication networks for training. In particular, we integrate two pairs of shared predictors for the model prediction in both server-to-client an…
Recently, blockchain-based federated learning (BFL) has attracted intensive research attention due to that the training process is auditable and the architecture is serverless avoiding the single point failure of the parameter server in vanilla federated learning (VFL). Nevertheless, BFL tremendously escalates the communication traffic volume because all local model updates (i.e., changes of model parameters) obtained by BFL clients will be transmitted to all miners for verification and to all clients for aggregation. In contrast, the parameter server and clients in VFL only retain aggregated …
We investigate the trade-off between rate, privacy and storage in federated learning (FL) with top $r$ sparsification, where the users and the servers in the FL system only share the most significant $r$ and $r'$ fractions, respectively, of updates and parameters in the FL process, to reduce the communication cost. We present schemes that guarantee information theoretic privacy of the values and indices of the sparse updates sent by the users at the expense of a larger storage cost. To this end, we generalize the scheme to reduce the storage cost by allowing a certain amount of information lea…
Federated Learning (FL) has received a significant amount of attention in the industry and research community due to its capability of keeping data on local devices. To aggregate the gradients of local models to train the global model, existing works require that the global model and the local models are the same. However, Internet of Things (IoT) devices are inherently diverse regarding computation speed and onboard memory. In this paper, we propose an FL framework targeting the heterogeneity of IoT devices. Specifically, local models are compressed from the global model, and the gradients of…
In federated learning (FL) with top $r$ sparsification, millions of users collectively train a machine learning (ML) model locally, using their personal data by only communicating the most significant $r$ fraction of updates to reduce the communication cost. It has been shown that the values as well as the indices of these selected (sparse) updates leak information about the users' personal data. In this work, we investigate different methods to carry out user-database communications in FL with top $r$ sparsification efficiently, while guaranteeing information theoretic privacy of users' perso…
Federated Learning (FL) solves many of this decade's concerns regarding data privacy and computation challenges. FL ensures no data leaves its source as the model is trained at where the data resides. However, FL comes with its own set of challenges. The communication of model weight updates in this distributed environment comes with significant network bandwidth costs. In this context, we propose a mechanism of compressing the weight updates using Autoencoders (AE), which learn the data features of the weight updates and subsequently perform compression. The encoder is set up on each of the n…
Compressed sensing is a signal processing technique whereby the limits imposed by the Shannon--Nyquist theorem can be exceeded provided certain conditions are imposed on the signal. Such conditions occur in many real-world scenarios, and compressed sensing has emerging applications in medical imaging, big data, and statistics. Finding practical matrix constructions and computationally efficient recovery algorithms for compressed sensing is an area of intense research interest. Many probabilistic matrix constructions have been proposed, and it is now well known that matrices with entries drawn …
Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；School of Computing, Faculty of Science and Engineering, Macquarie University（麦考瑞大学计算学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen University（深圳大学广东省人工智能与数字经济实验室）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）；School of Computing Science, Simon Fraser University（西蒙菲莎大学计算科学学院）
Distributed learning techniques such as federated learning have enabled multiple workers to train machine learning models together to reduce the overall training time. However, current distributed training algorithms (centralized or decentralized) suffer from the communication bottleneck on multiple low-bandwidth workers (also on the server under the centralized architecture). Although decentralized algorithms generally have lower communication complexity than the centralized counterpart, they still suffer from the communication bottleneck for workers with low network bandwidth. To deal with t…
