Adaptive optimization algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in some scenarios. However, recent studies show that they often lead to worse generalization performance than SGD, especially for training deep neural networks (DNNs). In this work, we identify the reasons that Adam generalizes worse than SGD, and develop a variant of Adam to eliminate the generalization gap. The proposed method, normalized direction-preserving Adam (ND-Adam), enables more precise control of the direction and step size for updating weight vectors, leading to significantly improved generalization performance. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also hope to shed light on why certain optimization algorithms generalize better than others.
Carnegie Mellon University（卡内基梅隆大学）；Beihang University（北京航空航天大学）；The University of Iowa（爱荷华大学）
In this paper, we propose a generic and simple strategy for utilizing stochastic gradient information in optimization. The technique essentially contains two consecutive steps in each iteration: 1) computing and normalizing each block (layer) of the mini-batch stochastic gradient; 2) selecting appropriate step size to update the decision variable (parameter) towards the negative of the block-normalized gradient. We conduct extensive empirical studies on various non-convex neural network optimization problems, including multi-layer perceptron, convolution neural networks and recurrent neural ne…
DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD
Intellifusion Inc.（Intellifusion公司）；Tsinghua University（清华大学）；Johns Hopkins University（约翰霍普金斯大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）
本文提出深度归一化 Transformer（DNT），通过在网络关键位置加入归一化来调节 Jacobian 矩阵并集中梯度分布，使普通 mSGDW 也能有效训练，且在 ViT 与 GPT 上表现更优。
Conda: Column-Normalized Adam for Training Large Language Models Faster
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（通用人工智能国家重点实验室，智能科学与技术学院，北京大学）；Singapore Management University（新加坡国立大学）；Nankai University（南开大学）；Beijing Normal University（北京师范大学）；ByteDance Seed（字节跳动种子）；Beijing University of Posts and Telecommunications（北京邮电大学）；Institute for Artificial Intelligence, Peking University（人工智能研究院，北京大学）；Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China（黄埔实验室（广州），广东，中国）
Amirkabir University of Technology（阿米拉比尔理工大学）；Ferdowsi University of Mashhad（马什哈德菲尔多西大学）
Many attempts took place to improve the adaptive filters that can also be useful to improve backpropagation (BP). Normalized least mean squares (NLMS) is one of the most successful algorithms derived from Least mean squares (LMS). However, its extension to multi-layer neural networks has not happened before. Here, we first show that it is possible to consider a multi-layer neural network as a stack of adaptive filters. Additionally, we introduce more comprehensible interpretations of NLMS than the complicated geometric interpretation in affine projection algorithm (APA) for a single fully-conn…
The development of machine learning is promoting the search for fast and stable minimization algorithms. To this end, we suggest a change in the current gradient descent methods that should speed up the motion in flat regions and slow it down in steep directions of the function to minimize. It is based on a "power gradient", in which each component of the gradient is replaced by its versus-preserving $H$-th power, with $0<H<1$. We test three modern gradient descent methods fed by such variant and by standard gradients, finding the new version to achieve significantly better performances for th…
Memory-Efficient LLM Pretraining via Minimalist Optimizer Design
Department of Electrical and Computer Engineering, University of Minnesota, USA（电气与计算机工程系，明尼苏达大学，美国）；School of Mathematics and Statistics, University of Sydney, Australia（数学与统计学学院，悉尼大学，澳大利亚）
Federated learning is a distributed and privacy-preserving approach to train a statistical model collaboratively from decentralized data of different parties. However, when datasets of participants are not independent and identically distributed (non-IID), models trained by naive federated algorithms may be biased towards certain participants, and model performance across participants is non-uniform. This is known as the fairness problem in federated learning. In this paper, we formulate fairness-controlled federated learning as a dynamical multi-objective optimization problem to ensure fair p…
Ant Group（蚂蚁集团）；Shanghai Jiao Tong University（上海交通大学）；University of Minnesota（明尼苏达大学）
The vanilla Differentially-Private Stochastic Gradient Descent (DP-SGD), including DP-Adam and other variants, ensures the privacy of training data by uniformly distributing privacy costs across training steps. The equivalent privacy costs controlled by maintaining the same gradient clipping thresholds and noise powers in each step result in unstable updates and a lower model accuracy when compared to the non-DP counterpart. In this paper, we propose the dynamic DP-SGD (along with dynamic DP-Adam, and others) to reduce the performance loss gap while maintaining privacy by dynamically adjusting…
National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）；NTT（日本电信电话株式会社）；University of St. Gallen（圣加仑大学）
Privacy-preserving machine learning (PPML) aims at enabling machine learning (ML) algorithms to be used on sensitive data. We contribute to this line of research by proposing a framework that allows efficient and secure evaluation of full-fledged state-of-the-art ML algorithms via secure multi-party computation (MPC). This is in contrast to most prior works, which substitute ML algorithms with approximated "MPC-friendly" variants. A drawback of the latter approach is that fine-tuning of the combined ML and MPC algorithms is required, which might lead to less efficient algorithms or inferior qu…
