In this paper, we present a distributed variant of adaptive stochastic gradient method for training deep neural networks in the parameter-server model. To reduce the communication cost among the workers and server, we incorporate two types of quantization schemes, i.e., gradient quantization and weight quantization, into the proposed distributed Adam. Besides, to reduce the bias introduced by quantization operations, we propose an error-feedback technique to compensate for the quantized gradient. Theoretically, in the stochastic nonconvex setting, we show that the distributed adaptive gradient method with gradient quantization and error-feedback converges to the first-order stationary point, and that the distributed adaptive gradient method with weight quantization and error-feedback converges to the point related to the quantized level under both the single-worker and multi-worker modes. At last, we apply the proposed distributed adaptive gradient methods to train deep neural networks. Experimental results demonstrate the efficacy of our methods.
Learning Accurate Decision Trees with Bandit Feedback via Quantized Gradient Descent
University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Berkeley（加州大学伯克利分校）；Microsoft Research, India（微软研究院印度分院）；Google Research, India（谷歌研究院印度分院）
Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization
Pillar of Information Systems Technology and Design（信息系统技术与设计学院）；School of Information Science and Engineering（信息科学与工程学院）；School of Cyber Science and Engineering（网络安全科学与工程学院）；National Mobile Communications Research Laboratory（国家移动通信研究实验室）；School of Electronic and Optical Engineering（电子与光学工程学院）；School of Electronics, Electrical Engineering and Computer Science (EEECS)（电子、电气工程与计算机科学学院）；Data61, CSIRO（Data61，CSIRO）；Department of Electronics Engineering（电子工程系）；Department of Electrical and Computer Engineering（电气与计算机工程系）
Quantitative Error Feedback for Quantization Noise Reduction of Filtering over Graphs
Division of Computer, Electrical and Mathematical Sciences and Engineering (CEMSE), King Abdullah University Of Science And Technology (KAUST)（计算机、电子和数学科学与工程系，国王阿卜杜勒-阿齐兹大学科学与技术（KAUST））；School of Information Science and Technology, ShanghaiTech University（信息科学与技术学院，上海科技大学）；Department of Electrical and Electronic Engineering, Imperial College London（电子与电气工程系，帝国理工学院伦敦分校）
Common-Mode Collapse and Recovery in Direct Feedback Alignment
Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University（哈佛大学肯普纳自然与人工智能研究所）；Howard Hughes Medical Institute（霍华德·休斯医学研究所）；Harvard Medical School（哈佛医学院）
Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy（中关村学院）；City University of Hong Kong（香港城市大学）；NJUST（南京理工大学）
