With the emergence of distributed data, training machine learning models in the serverless manner has attracted increasing attention in recent years. Numerous training approaches have been proposed in this regime, such as decentralized SGD. However, all existing decentralized algorithms only focus on standard SGD. It might not be suitable for some applications, such as deep factorization machine in which the feature is highly sparse and categorical so that the adaptive training algorithm is needed. In this paper, we propose a novel adaptive decentralized training approach, which can compute the learning rate from data dynamically. To the best of our knowledge, this is the first adaptive decentralized training approach. Our theoretical results reveal that the proposed algorithm can achieve linear speedup with respect to the number of workers. Moreover, to reduce the communication-efficient overhead, we further propose a communication-efficient adaptive decentralized training approach, which can also achieve linear speedup with respect to the number of workers. At last, extensive experiments on different tasks have confirmed the effectiveness of our proposed two approaches.
Towards Serverless Semi-Decentralized Federated Learning with Heterogeneous Optimizers
Department of Electrical and Computer Engineering, Princeton University（普林斯顿大学电子工程与计算机科学系）；Department of Electrical and Computer Engineering, Purdue University（帕克森大学电子工程与计算机科学系）
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（工业和信息化部模式分析与机器智能重点实验室）
Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization
Department of Mathematical Sciences（数学科学系）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；Department of Computer Science（计算机科学系）；MIT-IBM Watson AI Lab, IBM Research（MIT-IBM Watson AI实验室，IBM研究）；IBM T. J. Watson Research Center（IBM T.J. 华生研究中心）
Adaptive Graph Pruning with Sudden-Events Evaluation for Traffic Prediction using Online Semi-Decentralized ST-GNNs
University of Zagreb, Faculty of Electrical Engineering and Computing（Zagreb大学电子工程与计算学院）；RISE Research Institutes of Sweden（瑞典RISE研究机构）；KTH Royal Institute of Technology（皇家理工学院）
LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM T. J. Watson Research Center（IBM T. J. Watson 研究中心）；MIT-IBM Watson AI Lab, IBM Research（MIT-IBM Watson AI 实验室，IBM 研究院）
ADP-VRSGP: Decentralized Learning with Adaptive Differential Privacy via Variance-Reduced Stochastic Gradient Push
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences)（计算能力网络与信息安全重点实验室，教育部，山东计算机科学中心，齐鲁大学（山东科学院））；Shandong Provincial Key Laboratory of Industrial Network and Information System Security, Shandong Fundamental Research Center for Computer Science（山东省工业网络与信息系统安全重点实验室，山东省计算机科学基础研究中心）；School of Information and Software Engineering, University of Electronic Science and Technology of China（信息与软件工程学院，电子科技大学）
FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong Baptist University（香港浸会大学）；A*STAR（新加坡科技研究局）；Northeastern University（东北大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；XGRIDS；National University of Singapore（新加坡国立大学）
Dynamic Fee for Reducing Impermanent Loss in Decentralized Exchanges
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）；Faculty of Computer Science, HSE University（高等经济学院计算机学院）；Kharkevich Institute for Information Transmission Problems, RAS（俄罗斯科学院哈雷维奇信息传输问题研究所）
