In order to achieve the dual goals of privacy and learning across distributed data, Federated Learning (FL) systems rely on frequent exchanges of large files (model updates) between a set of clients and the server. As such FL systems are exposed to, or indeed the cause of, congestion across a wide set of network resources. Lossy compression can be used to reduce the size of exchanged files and associated delays, at the cost of adding noise to model updates. By judiciously adapting clients' compression to varying network congestion, an FL application can reduce wall clock training time. To that end, we propose a Network Adaptive Compression (NAC-FL) policy, which dynamically varies the client's lossy compression choices to network congestion variations. We prove, under appropriate assumptions, that NAC-FL is asymptotically optimal in terms of directly minimizing the expected wall clock training time. Further, we show via simulation that NAC-FL achieves robust performance improvements with higher gains in settings with positively correlated delays across time.
Federated learning (FL) is an emerging paradigm for training machine learning models using possibly private data available at edge devices. The distributed operation of FL gives rise to challenges that are not encountered in centralized machine learning, including the need to preserve the privacy of the local datasets, and the communication load due to the repeated exchange of updated models. These challenges are often tackled individually via techniques that induce some distortion on the updated models, e.g., local differential privacy (LDP) mechanisms and lossy compression. In this work we p…
Communication on heterogeneous edge networks is a fundamental bottleneck in Federated Learning (FL), restricting both model capacity and user participation. To address this issue, we introduce two novel strategies to reduce communication costs: (1) the use of lossy compression on the global model sent server-to-client; and (2) Federated Dropout, which allows users to efficiently train locally on smaller subsets of the global model and also provides a reduction in both client-to-server communication and local computation. We empirically show that these strategies, combined with existing compres…
VMware Research（威睿研究院）；University College London（伦敦大学学院）；Ben-Gurion University（本-古里安大学）；Stanford University（斯坦福大学）；Harvard University（哈佛大学）
Distributed Mean Estimation (DME) is a central building block in federated learning, where clients send local gradients to a parameter server for averaging and updating the model. Due to communication constraints, clients often use lossy compression techniques to compress the gradients, resulting in estimation inaccuracies. DME is more challenging when clients have diverse network conditions, such as constrained communication budgets and packet losses. In such settings, DME techniques often incur a significant increase in the estimation error leading to degraded learning performance. In this w…
In federated learning (FL), a machine learning model is trained on multiple nodes in a decentralized manner, while keeping the data local and not shared with other nodes. However, FL requires the nodes to also send information on the model parameters to a central server for aggregation. However, the information sent from the nodes to the server may reveal some details about each node's local data, thus raising privacy concerns. Furthermore, the repetitive uplink transmission from the nodes to the server may result in a communication overhead and network congestion. To address these two challen…
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）
Communicating information, like gradient vectors, between computing nodes in distributed and federated learning is typically an unavoidable burden, resulting in scalability issues. Indeed, communication might be slow and costly. Recent advances in communication-efficient training algorithms have reduced this bottleneck by using compression techniques, in the form of sparsification, quantization, or low-rank approximation. Since compression is a lossy, or inexact, process, the iteration complexity is typically worsened; but the total communication complexity can improve significantly, possibly …
Connected and Automated Vehicles (CAVs) are one of the emerging technologies in the automotive domain that has the potential to alleviate the issues of accidents, traffic congestion, and pollutant emissions, leading to a safe, efficient, and sustainable transportation system. Machine learning-based methods are widely used in CAVs for crucial tasks like perception, motion planning, and motion control, where machine learning models in CAVs are solely trained using the local vehicle data, and the performance is not certain when exposed to new environments or unseen conditions. Federated learning …
FedCova: Robust Federated Covariance Learning Against Noisy Labels
Department of Information Engineering（信息工程系）；The Chinese University of Hong Kong（香港中文大学）；National Key Laboratory on Wireless Communications（无线通信国家重点实验室）；University of Electronic Science and Technology of China（电子科学与技术大学）
Fraunhofer IVI（弗劳恩霍夫交通与基础设施系统研究所）；Technical University of Munich（慕尼黑工业大学）；Sony AI（索尼人工智能研究院）；Technische Hochschule Ingolstadt（英戈尔施塔特应用技术大学）
Federated learning enables cooperative training among massively distributed clients by sharing their learned local model parameters. However, with increasing model size, deploying federated learning requires a large communication bandwidth, which limits its deployment in wireless networks. To address this bottleneck, we introduce a residual-based federated learning framework (ResFed), where residuals rather than model parameters are transmitted in communication networks for training. In particular, we integrate two pairs of shared predictors for the model prediction in both server-to-client an…
Royal Institute of Technology (KTH)（皇家理工学院（KTH））；Princeton University（普林斯顿大学）
To leverage massive distributed data and computation resources, machine learning in the network edge is considered to be a promising technique especially for large-scale model training. Federated learning (FL), as a paradigm of collaborative learning techniques, has obtained increasing research attention with the benefits of communication efficiency and improved data privacy. Due to the lossy communication channels and limited communication resources (e.g., bandwidth and power), it is of interest to investigate fast responding and accurate FL schemes over wireless systems. Hence, we investigat…
In federated learning (FL), a global model is trained at a Parameter Server (PS) by aggregating model updates obtained from multiple remote learners. Generally, the communication between the remote users and the PS is rate-limited, while the transmission from the PS to the remote users are unconstrained. The FL setting gives rise to the distributed learning scenario in which the updates from the remote learners have to be compressed so as to meet communication rate constraints in the uplink transmission toward the PS. For this problem, one wishes to compress the model updates so as to minimize…
Princeton University（普林斯顿大学）；Imperial College London（伦敦帝国理工学院）
We study federated learning (FL), which enables mobile devices to utilize their local datasets to collaboratively train a global model with the help of a central server, while keeping data localized. At each iteration, the server broadcasts the current global model to the devices for local training, and aggregates the local model updates from the devices to update the global model. Previous work on the communication efficiency of FL has mainly focused on the aggregation of model updates from the devices, assuming perfect broadcasting of the global model. In this paper, we instead consider broa…
FedCau: A Proactive Stop Policy for Communication and Computation Efficient Federated Learning
School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（瑞典皇家理工学院电气与计算机科学学院）；Ericsson（爱立信公司）；Uppsala University（乌普萨拉大学）
Communication-efficient Vertical Federated Learning via Compressed Error Feedback
Carnegie Mellon University（卡内基梅隆大学）；Institute for Systems and Robotics（系统与机器人研究所）；Instituto Superior Técnico（里斯本高等理工学院）；NOVA School of Science and Technology（NOVA科学技术学院）
Towards Seamless Hierarchical Federated Learning under Intermittent Client Participation: A Stagewise Decision-Making Methodology
School of Informatics, Xiamen University（厦门大学信息学院）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海智能自主系统研究所）；School of Electronic Science and Engineering, Xiamen University（厦门大学电子科学与技术学院）；Department of Electrical Engineering, University at Buffalo–SUNY（布法罗大学（SUNY）电气工程系）；Department of Electrical and Computer Engineering, Western University（温哥华大学电气与计算机工程系）；Department of Electrical and Computer Engineering, North Carolina State University（北卡罗来纳州立大学电气与计算机工程系）；iF-Labs, Beijing Teleinfo Technology Co., Ltd.（北京 Teleinfo 技术有限公司 iF 实验室）
