Federated learning is a promising framework to mitigate data privacy and computation concerns. However, the communication cost between the server and clients has become the major bottleneck for successful deployment. Despite notable progress in gradient compression, the existing quantization methods require further improvement when low-bits compression is applied, especially the overall systems often degenerate a lot when quantization are applied in double directions to compress model weights and gradients. In this work, we propose a simple cosine-based nonlinear quantization and achieve impressive results in compressing round-trip communication costs. We are not only able to compress model weights and gradients at higher ratios than previous methods, but also achieve competing model performance at the same time. Further, our approach is highly suitable for federated learning problems since it has low computational complexity and requires only a little additional data to recover the compressed information. Extensive experiments have been conducted on image classification and brain tumor semantic segmentation using the CIFAR-10, and BraTS datasets where we show state-of-the-art effectiveness and impressive communication efficiency.
We explore the use of GPU for accelerating large scale nearest neighbor search and we propose a fast vector-quantization-based exhaustive nearest neighbor search algorithm that can achieve high accuracy without any indexing construction specifically designed for cosine similarity. This algorithm uses a novel XOR-friendly binary quantization method to encode floating-point numbers such that high-complexity multiplications can be optimized as low-complexity bitwise operations. Experiments show that, our quantization method takes short preprocessing time, and helps make the search speed of our ex…
PNCS:Power-Norm Cosine Similarity for Diverse Client Selection in Federated Learning
Department of Electrical and Computer Engineering, McMaster University, Hamilton, Canada（电气与计算机工程系，麦斯特大学）；Department of Electrical and Computer Engineering, National Yang-Ming Chiao-Tung University, Taiwan（电气与计算机工程系，国立阳明交通大学）
The Hong Kong University of Science and Technology（香港科技大学）
Federated learning (FL) is an emerging learning paradigm without violating users' privacy. However, large model size and frequent model aggregation cause serious communication bottleneck for FL. To reduce the communication volume, techniques such as model compression and quantization have been proposed. Besides the fixed-bit quantization, existing adaptive quantization schemes use ascending-trend quantization, where the quantization level increases with the training stages. In this paper, we first investigate the impact of quantization on model convergence, and show that the optimal quantizati…
Federated Learning (FL) is a powerful technique for training a model on a server with data from several clients in a privacy-preserving manner. In FL, a server sends the model to every client, who then train the model locally and send it back to the server. The server aggregates the updated models and repeats the process for several rounds. FL incurs significant communication costs, in particular when transmitting the updated local models from the clients back to the server. Recently proposed algorithms quantize the model parameters to efficiently compress FL communication. These algorithms ty…
Carnegie Mellon University（卡内基梅隆大学）；Weizmann Institute of Science（魏茨曼科学研究所）
Communication of model updates between client nodes and the central aggregating server is a major bottleneck in federated learning, especially in bandwidth-limited settings and high-dimensional models. Gradient quantization is an effective way of reducing the number of bits required to communicate each model update, albeit at the cost of having a higher error floor due to the higher variance of the stochastic gradients. In this work, we propose an adaptive quantization strategy called AdaQuantFL that aims to achieve communication efficiency as well as a low error floor by changing the number o…
Point Transformer with Federated Learning for Predicting Breast Cancer HER2 Status from Hematoxylin and Eosin-Stained Whole Slide Images
University of Science and Technology of China（中国科学技术大学）；Chinese Academy of Sciences（中国科学院）；West China Hospital, Sichuan University（四川大学华西医院）；Beihang University（北京航空航天大学）
The Chinese University of Hong Kong（香港中文大学）；Huawei Technologies Co. Ltd（华为技术有限公司）
The high cost of communicating gradients is a major bottleneck for federated learning, as the bandwidth of the participating user devices is limited. Existing gradient compression algorithms are mainly designed for data centers with high-speed network and achieve $O(\sqrt{d} \log d)$ per-iteration communication cost at best, where $d$ is the size of the model. We propose hyper-sphere quantization (HSQ), a general framework that can be configured to achieve a continuum of trade-offs between communication efficiency and gradient accuracy. In particular, at the high compression ratio end, HSQ pro…
This paper addresses the challenges of training large neural network models under federated learning settings: high on-device memory usage and communication cost. The proposed Online Model Compression (OMC) provides a framework that stores model parameters in a compressed format and decompresses them only when needed. We use quantization as the compression method in this paper and propose three methods, (1) using per-variable transformation, (2) weight matrices only quantization, and (3) partial parameter quantization, to minimize the impact on model accuracy. According to our experiments on t…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Recently, federated learning (FL) has sparked widespread attention as a promising decentralized machine learning approach which provides privacy and low delay. However, communication bottleneck still constitutes an issue, that needs to be resolved for an efficient deployment of FL over wireless networks. In this paper, we aim to minimize the total convergence time of FL, by quantizing the local model parameters prior to uplink transmission. More specifically, the convergence analysis of the FL algorithm with stochastic quantization is firstly presented, which reveals the impact of the quantiza…
UC Santa Barbara（加州大学圣巴巴拉分校）；UT Austin（德克萨斯大学奥斯汀分校）；UPenn（宾夕法尼亚大学）；MIT（麻省理工学院）
Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized. This framework faces several systems-oriented challenges which include (i) communication bottleneck since a large number of devices upload their local updates to a parameter server, and (ii) scalability as the federated network consists of millions of devices. Due to these systems challenges as well as issues related to statistical heterogeneity of data and privacy concerns, designing a provably efficient federated learning method is of significant importance…
University of Houston（休斯顿大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Science and Technology of China（中国科学技术大学）；University of Florida（佛罗里达大学）
Federated learning (FL) is a popular collaborative distributed machine learning paradigm across mobile devices. However, practical FL over resource constrained mobile devices confronts multiple challenges, e.g., the local on-device training and model updates in FL are power hungry and radio resource intensive for mobile devices. To address these challenges, in this paper, we attempt to take FL into the design of future wireless networks and develop a novel joint design of wireless transmission and weight quantization for energy efficient FL over mobile devices. Specifically, we develop flexibl…
University of Maryland, Baltimore County（马里兰大学巴尔的摩郡分校）
Federated Learning (FL) enables collaborative model building among a large number of participants without the need for explicit data sharing. But this approach shows vulnerabilities when privacy inference attacks are applied to it. In particular, in the event of a gradient leakage attack, which has a higher success rate in retrieving sensitive data from the model gradients, FL models are at higher risk due to the presence of communication in their inherent architecture. The most alarming thing about this gradient leakage attack is that it can be performed in such a covert way that it does not …
Federated learning (FL) is an emerging paradigm for training machine learning models using possibly private data available at edge devices. The distributed operation of FL gives rise to challenges that are not encountered in centralized machine learning, including the need to preserve the privacy of the local datasets, and the communication load due to the repeated exchange of updated models. These challenges are often tackled individually via techniques that induce some distortion on the updated models, e.g., local differential privacy (LDP) mechanisms and lossy compression. In this work we p…
Ben-Gurion University of the Negev（内盖夫本-古里安大学）；Princeton University（普林斯顿大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Weizmann Institute of Science（魏茨曼科学研究所）
Traditional deep learning models are trained at a centralized server using labeled data samples collected from end devices or users. Such data samples often include private information, which the users may not be willing to share. Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their possibly private labeled data. In FL, each user trains its copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. A major challenge that arises in this method is the need of each…
Xiamen University（厦门大学）；University of Melbourne（墨尔本大学）
Fairness has emerged as a critical problem in federated learning (FL). In this work, we identify a cause of unfairness in FL -- conflicting gradients with large differences in the magnitudes. To address this issue, we propose the federated fair averaging (FedFV) algorithm to mitigate potential conflicts among clients before averaging their gradients. We first use the cosine similarity to detect gradient conflicts, and then iteratively eliminate such conflicts by modifying both the direction and the magnitude of the gradients. We further show the theoretical foundation of FedFV to mitigate the …
FedShift: Robust Federated Learning Aggregation Scheme in Resource Constrained Environment via Weight Shifting
Department of Electrical Engineering and Computer Science, University of Stavanger（电气工程与计算机科学系，斯塔万格大学）；Department of Computer Convergence Software, Korea University（计算机融合软件系，韩国大学）
Enhancing Convergence, Privacy and Fairness for Wireless Personalized Federated Learning: Quantization-Assisted Min-Max Fair Scheduling
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（信息与通信工程学院，北京邮电大学）；School of Computing, Macquarie University（计算学院，麦考瑞大学）；Department of Broadband Communication, Peng Cheng Laboratory（宽带通信部，鹏城实验室）；China Telecom, Sichuan Branch（中国电信四川分公司）
AQUILA: Communication Efficient Federated Learning with Adaptive Quantization in Device Selection Strategy
Tsinghua University（清华大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Shanghai AI Lab（上海人工智能实验室）；George Washington University（乔治华盛顿大学）；Ryerson University（瑞尔森大学）
FedX: Adaptive Model Decomposition and Quantization for IoT Federated Learning
University at Albany（阿巴拉契亚大学）；Southern Illinois University（南方伊利诺伊大学）；New Jersey Institute of Technology（新泽西理工学院）；Qualcomm Incorporated（高通公司）；Kent State University（凯斯西储大学）
Sun Yat-sen University（中山大学）；University of Virginia（弗吉尼亚大学）
Communication has been known to be one of the primary bottlenecks of federated learning (FL), and yet existing studies have not addressed the efficient communication design, particularly in wireless FL where both uplink and downlink communications have to be considered. In this paper, we focus on the design and analysis of physical layer quantization and transmission methods for wireless FL. We answer the question of what and how to communicate between clients and the parameter server and evaluate the impact of the various quantization and transmission options of the updated model on the learn…
Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face are heterogeneity in data across clients and collaboration of clients with {\em diverse resources}. In this work, we introduce a \textit{quantized} and \textit{personalized} FL algorithm QuPeL that facilitates collective training with heterogeneous clients while respecting resource diversity. For personalization, we allow clients to learn \textit{compressed personalized models} with different quantization parameters…
Cross-device Federated Learning is an increasingly popular machine learning setting to train a model by leveraging a large population of client devices with high privacy and security guarantees. However, communication efficiency remains a major bottleneck when scaling federated learning to production environments, particularly due to bandwidth constraints during uplink communication. In this paper, we formalize and address the problem of compressing client-to-server model updates under the Secure Aggregation primitive, a core component of Federated Learning pipelines that allows the server to …
Performance Optimization for Variable Bitwidth Federated Learning in Wireless Networks
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Miami（迈阿密大学）；Purdue University（普渡大学）；Virginia Tech（弗吉尼亚理工大学）；The Chinese University of Hong Kong（香港中文大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Peng Cheng Laboratory（鹏城实验室）
East China University of Science and Technology（华东理工大学）；Tongji University（同济大学）；University of Surrey（萨里大学）；Southern University of Science and Technology（南方科技大学）
Learning over massive data stored in different locations is essential in many real-world applications. However, sharing data is full of challenges due to the increasing demands of privacy and security with the growing use of smart mobile devices and IoT devices. Federated learning provides a potential solution to privacy-preserving and secure machine learning, by means of jointly training a global model without uploading data distributed on multiple devices to a central server. However, most existing work on federated learning adopts machine learning models with full-precision weights, and alm…
Federated Learning of Binary Neural Networks: Enabling Low-Cost Inference
Department of Electrical Engineering, Indian Institute of Technology Madras（印度理工学院马德拉斯分校电子工程系）；Department of Computer Science and Engineering, Jadavpur University（贾达布尔大学计算机科学与工程系）
Privacy-Preserving Quantized Federated Learning with Diverse Precision
Department of Electrical Engineering and Computer Science, The University of Kansas（堪萨斯大学电气工程与计算机科学系）；Department of Information and Communications Engineering, Aalto University（阿alto大学信息与通信工程系）；School of Electrical and Computer Engineering, Purdue University（普渡大学电气与计算机工程学院）；School of Electrical, Computer and Energy Engineering, Arizona State University（亚利桑那州立大学电气、计算机与能源工程学院）
Federated learning is widely used to perform decentralized training of a global model on multiple devices while preserving the data privacy of each device. However, it suffers from heterogeneous local data on each training device which increases the difficulty to reach the same level of accuracy as the centralized training. Supervised Contrastive Learning which outperform cross-entropy tries to minimizes the difference between feature space of points belongs to the same class and pushes away points from different classes. We propose Supervised Contrastive Federated Learning in which devices ca…
In this work, we study the problem of federated learning (FL), where distributed users aim to jointly train a machine learning model with the help of a parameter server (PS). In each iteration of FL, users compute local gradients, followed by transmission of the quantized gradients for subsequent aggregation and model updates at PS. One of the challenges of FL is that of communication overhead due to FL's iterative nature and large model sizes. One recent direction to alleviate communication bottleneck in FL is to let users communicate simultaneously over a multiple access channel (MAC), possi…
Efficient Asynchronous Federated Learning with Sparsification and Quantization
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Hithink RoyalFlush Information Network Co., Ltd.（同花顺信息服务股份有限公司）；Collaborative Innovation Center of Novel Software Technology and Industrialization（新型软件技术及产业化协同创新中心）；Department of Sciences and Informatics, Muroran Institute of Technology（室兰工业大学科学与信息学系）；Boston Consulting Group（波士顿咨询公司）
Deep learning models have been successfully used in computer vision and many other fields. We propose an unorthodox algorithm for performing quantization of the model parameters. In contrast with popular quantization schemes based on thresholds, we use a novel technique based on periodic functions, such as continuous trigonometric sine or cosine as well as non-continuous hat functions. We apply these functions component-wise and add the sum over the model parameters as a regularizer to the model loss during training. The frequency and amplitude hyper-parameters of these functions can be adjust…
Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face are heterogeneity in data across clients and collaboration of clients with {\em diverse resources}. In this work, we introduce a \textit{quantized} and \textit{personalized} FL algorithm QuPeD that facilitates collective (personalized model compression) training via \textit{knowledge distillation} (KD) among clients who have access to heterogeneous data and resources. For personalization, we allow clients to learn \…
Lightweight Federated Learning over Wireless Edge Networks
Department of Electronic Engineering, Tsinghua University（电子工程系，清华大学）；School of Cyber Science and Technology, Beihang University（信息科学与技术学院，北航）；Tsinghua Space Center, Tsinghua University（清华大学空间中心）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）
