Existing approaches to federated learning suffer from a communication bottleneck as well as convergence issues due to sparse client participation. In this paper we introduce a novel algorithm, called FetchSGD, to overcome these challenges. FetchSGD compresses model updates using a Count Sketch, and then takes advantage of the mergeability of sketches to combine model updates from many workers. A key insight in the design of FetchSGD is that, because the Count Sketch is linear, momentum and error accumulation can both be carried out within the sketch. This allows the algorithm to move momentum and error accumulation from clients to the central aggregator, overcoming the challenges of sparse client participation while still achieving high compression rates and good convergence. We prove that FetchSGD has favorable convergence guarantees, and we demonstrate its empirical effectiveness by training two residual networks and a transformer model.
FLECS-CGD: A Federated Learning Second-Order Framework via Compression and Sketching with Compressed Gradient Differences
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）
在最近的论文FLECS（Agafonov et al，FLECS：A联邦学习Second-Order Framework via Compression and Sketching）中，针对联邦学习问题提出了二阶框架FLECS。该方法利用Hessian草图的压缩，使通信成本低。然而，FLECS的主要瓶颈是没有压缩的梯度通信。在本文中，我们提出了修改的FLECS压缩梯度差异，我们称之为FLECS-CGD（FLECS压缩梯度差异），使其适用于随机优化。在强凸和非凸的情况下提供收敛保证。实验结果表明，该方法具有较好的实用价值.
Communication-Efficient Personalized Federated Learning via Layer-Wise Multi-Threshold Random Sketching
School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；Faculty of Infor-X, Xidian University（西安电子科技大学Infor-X学院）；College of Artificial Intelligence, Inner Mongolia Normal University（内蒙古师范大学人工智能学院）
Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Sampling
National Engineering Research Center for Mobile Network Technologies, Beijing University of Posts and Telecommunications（移动网络技术国家工程研究中心，北京邮电大学）；Hong Kong Baptist University（香港 Baptist大学）；Peng Cheng Laboratory（鹏城实验室）
