Scalable training of large models (like BERT and GPT-3) requires careful optimization rooted in model design, architecture, and system capabilities. From a system standpoint, communication has become a major bottleneck, especially on commodity systems with standard TCP interconnects that offer limited network bandwidth. Communication compression is an important technique to reduce training time on such systems. One of the most effective methods is error-compensated compression, which offers robust convergence speed even under 1-bit compression. However, state-of-the-art error compensation techniques only work with basic optimizers like SGD and momentum SGD, which are linearly dependent on the gradients. They do not work with non-linear gradient-based optimizers like Adam, which offer state-of-the-art convergence efficiency and accuracy for models like BERT. In this paper, we propose 1-bit Adam that reduces the communication volume by up to $5\times$, offers much better scalability, and provides the same convergence speed as uncompressed Adam. Our key finding is that Adam's variance (non-linear term) becomes stable (after a warmup phase) and can be used as a fixed precondition for the rest of the training (compression phase). Experiments on up to 256 GPUs show that 1-bit Adam enables up to $3.3\times$ higher throughput for BERT-Large pre-training and up to $2.9\times$ higher throughput for SQuAD fine-tuning. In addition, we provide theoretical analysis for our proposed work.
Distributed data mining is an emerging research topic to effectively and efficiently address hard data mining tasks using big data, which are partitioned and computed on different worker nodes, instead of one centralized server. Nevertheless, distributed learning methods often suffer from the communication bottleneck when the network bandwidth is limited or the size of model is large. To solve this critical issue, many gradient compression methods have been proposed recently to reduce the communication cost for multiple optimization algorithms. However, the current applications of gradient com…
Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM
University of Connecticut（康涅狄格大学）
自适应梯度方法（AGM）在深度学习领域的非凸优化问题中得到了广泛的应用。我们重新审视AGM，并确定AGM使用的自适应学习率（A-LR）在不同时期的问题维度上变化很大（即，各向异性尺度），这可能导致收敛和泛化方面的问题。所有现有的修改后年度股东大会实际上都代表了修改A-LR的努力。在理论上，我们提供了一种新的方法来分析AGM的收敛性，并证明了\textsc{Adam}的收敛速度也依赖于它的超参数$ε$，这是以前被忽视的。基于这两个事实，我们提出了一个新的AGM校准A-LR与激活（{ \em softplus}）功能，导致\textsc{Sadam}和\textsc{SAMSGrad}方法\footnote{Code is available at https://github.com/neilliang90/Sadam.git.}。我们进一步证明了这些算法在非凸、非强凸和Polyak-Projojasiewicz条件下比\textsc{Adam}具有更好的收敛速度。实证研究支持我们对各向异性A-LR的观察，并表明所提出的方法优于现有的AGM，并且在多个深度学习任务中比S-Momentum更好地推广。
A New Adaptive Gradient Method with Gradient Decomposition
Center for Data Science（数据科学中心）；Peking University（北京大学）；Key Lab. of Machine Perception (MOE)（机器感知教育部重点实验室）；School of EECS, Peking University（北京大学电子工程与计算机科学学院）；Peng Cheng Laboratory（鹏城实验室）
Network quantization is a dominant paradigm of model compression. However, the abrupt changes in quantized weights during training often lead to severe loss fluctuations and result in a sharp loss landscape, making the gradients unstable and thus degrading the performance. Recently, Sharpness-Aware Minimization (SAM) has been proposed to smooth the loss landscape and improve the generalization performance of the models. Nevertheless, directly applying SAM to the quantized models can lead to perturbation mismatch or diminishment issues, resulting in suboptimal performance. In this paper, we pro…
FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models
School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）；Medical College, Tianjin University（天津大学医学院）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院通用人工智能国家重点实验室）；Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China（广州黄埔 Pazhou 实验室）
Conda: Column-Normalized Adam for Training Large Language Models Faster
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（通用人工智能国家重点实验室，智能科学与技术学院，北京大学）；Singapore Management University（新加坡国立大学）；Nankai University（南开大学）；Beijing Normal University（北京师范大学）；ByteDance Seed（字节跳动种子）；Beijing University of Posts and Telecommunications（北京邮电大学）；Institute for Artificial Intelligence, Peking University（人工智能研究院，北京大学）；Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China（黄埔实验室（广州），广东，中国）
SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
University of Exeter（埃克塞特大学）；Eindhoven University of Technology（埃因霍温理工大学）；University of Leicester（莱斯特大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Oxford（牛津大学）
LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM T. J. Watson Research Center（IBM T. J. Watson 研究中心）；MIT-IBM Watson AI Lab, IBM Research（MIT-IBM Watson AI 实验室，IBM 研究院）
GWT: Scalable Optimizer State Compression for Large Language Model Training
National Key Laboratory of Parallel and Distributed Computing（并行与分布式计算国家重点实验室）；National University of Defense Technology（国防科技大学）；Center for Machine Learning Research（机器学习研究中心）
The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；Shanghai Jiao Tong University（上海交通大学）；AI for Science Institute, Beijing, China（北京人工智能科学研究院）
