Minibatching Offers Improved Generalization Performance for Second Order Optimizers
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Washington State University（华盛顿州立大学）；Rutgers, The State University of New Jersey（罗格斯新泽西州立大学）；University of Washington（华盛顿大学）；Pacific Northwest National Laboratory（太平洋西北国家实验室）
Eric Silk, Swarnita Chakraborty, Nairanjana Dasgupta, Anand D. Sarwate, Andrew Lumsdaine, Tony Chiang
Training deep neural networks (DNNs) used in modern machine learning is computationally expensive. Machine learning scientists, therefore, rely on stochastic first-order methods for training, coupled with significant hand-tuning, to obtain good performance. To better understand performance variability of different stochastic algorithms, including second-order methods, we conduct an empirical study that treats performance as a response variable across multiple training sessions of the same model. Using 2-factor Analysis of Variance (ANOVA) with interactions, we show that batch size used during training has a statistically significant effect on the peak accuracy of the methods, and that full batch largely performed the worst. In addition, we found that second-order optimizers (SOOs) generally exhibited significantly lower variance at specific batch sizes, suggesting they may require less hyperparameter tuning, leading to a reduced overall time to solution for model training.
An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning
1 Department of Computer Science \& Engineering, Indian Institute of Technology Indore, India.；3 National Remote Sensing Centre, Indian Space Research Organisation, India.
Adversarial attack has recently become a tremendous threat to deep learning models. To improve the robustness of machine learning models, adversarial training, formulated as a minimax optimization problem, has been recognized as one of the most effective defense mechanisms. However, the non-convex and non-concave property poses a great challenge to the minimax training. In this paper, we empirically demonstrate that the commonly used PGD attack may not be optimal for inner maximization, and improved inner optimizer can lead to a more robust model. Then we leverage a learning-to-learn (L2L) fra…
MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（信息科技部模式分析与机器智能重点实验室）；College of Design and Engineering, National University of Singapore（新加坡国立大学设计与工程学院）
$μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
Université de Montréal（蒙特利尔大学）；Mila – Quebec AI Institute（魁北克人工智能研究所）；Concordia University（康科迪亚大学）；Samsung AI Lab（三星AI实验室）；ISIR, Sorbonne University, CNRS, Paris, France（巴黎法国索邦大学ISIR研究所、法国国家科学研究中心）
State-of-the art vision models can achieve superhuman performance on image classification tasks when testing and training data come from the same distribution. However, when models are tested on corrupted images (e.g. due to scale changes, translations, or shifts in brightness or contrast), performance degrades significantly. Here, we explore the possibility of meta-training a learned optimizer that can train image classification models such that they are robust to common image corruptions. Specifically, we are interested training models that are more robust to noise distributions not present …
Learned optimizers -- neural networks that are trained to act as optimizers -- have the potential to dramatically accelerate training of machine learning models. However, even when meta-trained across thousands of tasks at huge computational expense, blackbox learned optimizers often struggle with stability and generalization when applied to tasks unlike those in their meta-training set. In this paper, we use tools from dynamical systems to investigate the inductive biases and stability properties of optimization algorithms, and apply the resulting insights to designing inductive biases for bl…
Here I present a small update to the bias-correction term in the Adam optimizer that has the advantage of making smaller gradient updates in the first several steps of training. With the default bias-correction, Adam may actually make larger than requested gradient updates early in training. By only including the well-justified bias-correction of the second moment gradient estimate, $v_t$, and excluding the bias-correction on the first-order estimate, $m_t$, we attain these more desirable gradient update properties in the first series of steps. The default implementation of Adam may be as sens…
University of Chicago（芝加哥大学）；University of Texas at Austin（得克萨斯大学奥斯汀分校）；Texas Advanced Computing Center（得克萨斯高级计算中心）；University of Wisconsin, Madison（威斯康星大学麦迪逊分校）；Argonne National Laboratory（阿贡国家实验室）
Kronecker-factored Approximate Curvature (K-FAC) has recently been shown to converge faster in deep neural network (DNN) training than stochastic gradient descent (SGD); however, K-FAC's larger memory footprint hinders its applicability to large models. We present KAISA, a K-FAC-enabled, Adaptable, Improved, and ScAlable second-order optimizer framework that adapts the memory footprint, communication, and computation given specific models and hardware to improve performance and increase scalability. We quantify the tradeoffs between memory and communication cost and evaluate KAISA on large mod…
Adaptive Regularization for Sparsity Control in Bregman-Based Optimizers
Department of Data Science, FAU Erlangen-Nürnberg（FAU厄林根-纽伦堡数据科学系）；International Audiolabs, FAU Erlangen-Nürnberg（FAU厄林根-纽伦堡国际声学实验室）；School of Computation, Information and Technology, Technical University of Munich（慕尼黑技术大学计算、信息与技术学院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
The Ohio State University（俄亥俄州立大学）；UT Austin（德克萨斯大学奥斯汀分校）；JD Explore Academy（京东探索研究院）
Learning to optimize (L2O) has gained increasing popularity, which automates the design of optimizers by data-driven approaches. However, current L2O methods often suffer from poor generalization performance in at least two folds: (i) applying the L2O-learned optimizer to unseen optimizees, in terms of lowering their loss function values (optimizer generalization, or ``generalizable learning of optimizers"); and (ii) the test performance of an optimizee (itself as a machine learning model), trained by the optimizer, in terms of the accuracy over unseen data (optimizee generalization, or ``lear…
Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness
Department of Electrical and Computer Engineering, Queen’s University（皇后大学电气与计算机工程系）；Department of Computer Science and Technology, University of Cambridge（剑桥大学计算机科学与技术系）
GWT: Scalable Optimizer State Compression for Large Language Model Training
National Key Laboratory of Parallel and Distributed Computing（并行与分布式计算国家重点实验室）；National University of Defense Technology（国防科技大学）；Center for Machine Learning Research（机器学习研究中心）
Momentum is a widely used technique for gradient-based optimizers in deep learning. In this paper, we propose a decaying momentum (\textsc{Demon}) rule. We conduct the first large-scale empirical analysis of momentum decay methods for modern neural network optimization, in addition to the most popular learning rate decay schedules. Across 28 relevant combinations of models, epochs, datasets, and optimizers, \textsc{Demon} achieves the highest number of Top-1 and Top-3 finishes at 39\% and 85\% respectively, almost doubling the second-placed learning rate cosine schedule at 17\% and 60\%, respe…
Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）；Institute of High Performance Computing (IHPC), A*STAR（新加坡科技研究局高性能计算研究所）
Overparametrized Deep Neural Networks (DNNs) often achieve astounding performances, but may potentially result in severe generalization error. Recently, the relation between the sharpness of the loss landscape and the generalization error has been established by Foret et al. (2020), in which the Sharpness Aware Minimizer (SAM) was proposed to mitigate the degradation of the generalization. Unfortunately, SAM s computational cost is roughly double that of base optimizers, such as Stochastic Gradient Descent (SGD). This paper thus proposes Efficient Sharpness Aware Minimizer (ESAM), which boosts…
We study the factors affecting training time in multi-device deep learning systems. Given a specification of a convolutional neural network, our goal is to minimize the time to train this model on a cluster of commodity CPUs and GPUs. We first focus on the single-node setting and show that by using standard batching and data-parallel techniques, throughput can be improved by at least 5.5x over state-of-the-art systems on CPUs. This ensures an end-to-end training speed directly proportional to the throughput of a device regardless of its underlying hardware, allowing each node in the cluster to…
Czech Technical University in Prague（布拉格捷克理工大学）；GoodAI
Designing faster optimization algorithms is of ever-growing interest. In recent years, learning to learn methods that learn how to optimize demonstrated very encouraging results. Current approaches usually do not effectively include the dynamics of the optimization process during training. They either omit it entirely or only implicitly assume the dynamics of an isolated parameter. In this paper, we show how to utilize the dynamic mode decomposition method for extracting informative features about optimization dynamics. By employing those features, we show that our learned optimizer generalize…
University of Science and Technology of China（中国科学技术大学）；Duke Kunshan University（昆山杜克大学）；Edinburgh Napier University（爱丁堡龙比亚大学）；Gusu Laboratory of Materials（姑苏材料实验室）
AdaBelief, one of the current best optimizers, demonstrates superior generalization ability compared to the popular Adam algorithm by viewing the exponential moving average of observed gradients. AdaBelief is theoretically appealing in that it has a data-dependent $O(\sqrt{T})$ regret bound when objective functions are convex, where $T$ is a time horizon. It remains however an open problem whether the convergence rate can be further improved without sacrificing its generalization ability. %on how to exploit strong convexity to further improve the convergence rate of AdaBelief. To this end, we …
Salzburg University of Applied Sciences（萨尔茨堡应用科学大学）；Paris Lodron University Salzburg（帕拉塞尔苏斯大学萨尔茨堡）；Josef Ressel Centre for Intelligent and Secure Industrial Automation（约瑟夫·雷塞尔智能与安全工业自动化中心）
Muon with Spectral Guidance: Efficient Optimization for Scientific Machine Learning
Elmore Family School of Electrical and Computer Engineering（埃洛姆家族电气与计算机工程学院）；Purdue University（普渡大学）；Department of Mathematics（数学系）；Department of Mathematics & School of Mechanical Engineering（数学系与机械工程学院）
Second-order optimizers hold intriguing potential for deep learning, but suffer from increased cost and sensitivity to the non-convexity of the loss surface as compared to gradient-based approaches. We introduce a coordinate descent method to train deep neural networks for classification tasks that exploits global convexity of the cross-entropy loss in the weights of the linear layer. Our hybrid Newton/Gradient Descent (NGD) method is consistent with the interpretation of hidden layers as providing an adaptive basis and the linear layer as providing an optimal fit of the basis to data. By alte…
University of Science and Technology of China（中国科学技术大学）；MIT-IBM Watson AI Lab, IBM Research（麻省理工学院-IBM Watson人工智能实验室，IBM研究）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Shanghai Jiao Tong University（上海交通大学）
Learning to optimize (L2O) has gained increasing attention since classical optimizers require laborious problem-specific design and hyperparameter tuning. However, there is a gap between the practical demand and the achievable performance of existing L2O models. Specifically, those learned optimizers are applicable to only a limited class of problems, and often exhibit instability. With many efforts devoted to designing more sophisticated L2O models, we argue for another orthogonal, under-explored theme: the training techniques for those L2O models. We show that even the simplest L2O model cou…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
We introduce optimization methods for convolutional neural networks that can be used to improve existing gradient-based optimization in terms of generalization error. The method requires only simple processing of existing stochastic gradients, can be used in conjunction with any optimizer, and has only a linear overhead (in the number of parameters) compared to computation of the stochastic gradient. The method works by computing the gradient of the loss function with respect to output-channel directed re-weighted L2 or Sobolev metrics, which has the effect of smoothing components of the gradi…
Deep learning practitioners often operate on a computational and monetary budget. Thus, it is critical to design optimization algorithms that perform well under any budget. The linear learning rate schedule is considered the best budget-aware schedule, as it outperforms most other schedules in the low budget regime. On the other hand, learning rate schedules -- such as the \texttt{30-60-90} step schedule -- are known to achieve high performance when the model can be trained for many epochs. Yet, it is often not known a priori whether one's budget will be large or small; thus, the optimal choic…
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
University of Waterloo（滑铁卢大学）；University of California, Berkeley（加州大学伯克利分校）；ETH Zurich（苏黎世联邦理工学院）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Michigan（密歇根大学）
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；School of Management, Xi’an Jiaotong University（西安交通大学管理学院）；School of Software, Xi’an Jiaotong University（西安交通大学软件学院）
Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）；ByteDance（字节跳动）
Modern deep neural networks (DNNs) have achieved state-of-the-art performances but are typically over-parameterized. The over-parameterization may result in undesirably large generalization error in the absence of other customized training strategies. Recently, a line of research under the name of Sharpness-Aware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. However, SAM-like methods incur a two-fold computational overhead of the given base optimizer (e.g. SGD) for approxim…
Conda: Column-Normalized Adam for Training Large Language Models Faster
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（通用人工智能国家重点实验室，智能科学与技术学院，北京大学）；Singapore Management University（新加坡国立大学）；Nankai University（南开大学）；Beijing Normal University（北京师范大学）；ByteDance Seed（字节跳动种子）；Beijing University of Posts and Telecommunications（北京邮电大学）；Institute for Artificial Intelligence, Peking University（人工智能研究院，北京大学）；Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China（黄埔实验室（广州），广东，中国）
Machine learning techniques applied to software engineering tasks can be improved by hyperparameter optimization, i.e., automatic tools that find good settings for a learner's control parameters. We show that such hyperparameter optimization can be unnecessarily slow, particularly when the optimizers waste time exploring "redundant tunings"', i.e., pairs of tunings which lead to indistinguishable results. By ignoring redundant tunings, DODGE, a tuning tool, runs orders of magnitude faster, while also generating learners with more accurate predictions than seen in prior state-of-the-art approac…
First-Passage Approach to Optimizing Perturbations for Improved Training of Machine Learning Models
School of Chemistry, Tel Aviv University（化学系，特拉维夫大学）；The Center for Physics and Chemistry of Living Systems, Tel Aviv University（生命系统物理与化学中心，特拉维夫大学）；The Center for Computational Molecular and Materials Science, Tel Aviv University（计算分子与材料科学中心，特拉维夫大学）
University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
Generating images from natural language instructions is an intriguing yet highly challenging task. We approach text-to-image generation by combining the power of the retrained CLIP representation with an off-the-shelf image generator (GANs), optimizing in the latent space of GAN to find images that achieve maximum CLIP score with the given input text. Compared to traditional methods that train generative models from text to image starting from scratch, the CLIP+GAN approach is training-free, zero shot and can be easily customized with different generators. However, optimizing CLIP score in the…
