Federated Learning (FL) is a promising privacy-preserving distributed learning paradigm but suffers from high communication cost when training large-scale machine learning models. Sign-based methods, such as SignSGD \cite{bernstein2018signsgd}, have been proposed as a biased gradient compression technique for reducing the communication cost. However, sign-based algorithms could diverge under heterogeneous data, which thus motivated the development of advanced techniques, such as the error-feedback method and stochastic sign-based compression, to fix this issue. Nevertheless, these methods still suffer from slower convergence rates. Besides, none of them allows multiple local SGD updates like FedAvg \cite{mcmahan2017communication}. In this paper, we propose a novel noisy perturbation scheme with a general symmetric noise distribution for sign-based compression, which not only allows one to flexibly control the tradeoff between gradient bias and convergence performance, but also provides a unified viewpoint to existing stochastic sign-based methods. More importantly, the unified noisy perturbation scheme enables the development of the very first sign-based FedAvg algorithm ($z$-SignFedAvg) to accelerate the convergence. Theoretically, we show that $z$-SignFedAvg achieves a faster convergence rate than existing sign-based methods and, under the uniformly distributed noise, can enjoy the same convergence rate as its uncompressed counterpart. Extensive experiments are conducted to demonstrate that the $z$-SignFedAvg can achieve competitive empirical performance on real datasets and outperforms existing schemes.
Better Methods and Theory for Federated Learning: Compression, Client Selection and Heterogeneity
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
联邦学习（FL）是涉及多个客户端的新兴机器学习范例，例如，移动电话设备，具有协作解决由中央服务器协调的机器学习问题的激励。FL是由Konečnet al.和McMahan et al.在2016年提出的，作为传统集中式机器学习的一种可行的隐私保护替代方案，因为通过构造，训练数据点是分散的，并且永远不会由客户端传输到中央服务器。因此，FL在一定程度上减轻了与集中式数据汇集相关的隐私风险。不幸的是，FL的优化面临着集中式优化通常不需要处理的几个特定问题。在这篇论文中，我们确定了这些挑战中的一些，并提出了新的方法和算法来解决这些问题，最终目标是使实际的FL解决方案支持数学上严格的保证。
FLECS-CGD: A Federated Learning Second-Order Framework via Compression and Sketching with Compressed Gradient Differences
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）
在最近的论文FLECS（Agafonov et al，FLECS：A联邦学习Second-Order Framework via Compression and Sketching）中，针对联邦学习问题提出了二阶框架FLECS。该方法利用Hessian草图的压缩，使通信成本低。然而，FLECS的主要瓶颈是没有压缩的梯度通信。在本文中，我们提出了修改的FLECS压缩梯度差异，我们称之为FLECS-CGD（FLECS压缩梯度差异），使其适用于随机优化。在强凸和非凸的情况下提供收敛保证。实验结果表明，该方法具有较好的实用价值.
Wyner-Ziv Gradient Compression for Federated Learning
ShanghaiTech University（上海科技大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Ternary Compression for Communication-Efficient Federated Learning
East China University of Science and Technology（华东理工大学）；Tongji University（同济大学）；University of Surrey（萨里大学）；Southern University of Science and Technology（南方科技大学）
Optimal Rate Adaption in Federated Learning with Compressed Communications
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；School of Computing, Faculty of Science and Engineering, Macquarie University（麦考瑞大学科学与工程学院计算机学院）；Peng Cheng Laboratory（鹏城实验室）；School of Computing Science, Simon Fraser University（西蒙弗雷泽大学计算科学学院）
A Joint Approach to Local Updating and Gradient Compression for Efficient Asynchronous Federated Learning
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学伯克利深圳研究院）；Southern University of Science and Technology（南方科技大学）；School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机科学与技术学院）
Expediting In-Network Federated Learning by Voting-Based Consensus Model Compression
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；School of Computing, Faculty of Science and Engineering, Macquarie University（麦考瑞大学科学与工程学院计算学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen University（深圳大学广东省人工智能与数字经济实验室（深圳））；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
Efficient Model Compression for Hierarchical Federated Learning
School of Systems Science and Engineering, Sun Yat-Sen University（中山大学系统科学与工程学院）；School of Intelligent Systems Engineering, Sun Yat-Sen University（中山大学智能工程学院）
Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong Baptist University（香港浸会大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
Federated Optimization Algorithms with Random Reshuffling and Gradient Compression
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Mila, Université de Montréal（蒙特利尔大学米拉研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Princeton University（普林斯顿大学）
Caesar: A Low-deviation Compression Approach for Efficient Federated Learning
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Guangdong OPPO Mobile Telecommunications Corp., Ltd.（广东欧珀移动通信有限公司）
