Training neural networks with large batch is of fundamental significance to deep learning. Large batch training remarkably reduces the amount of training time but has difficulties in maintaining accuracy. Recent works have put forward optimization methods such as LARS and LAMB to tackle this issue through adaptive layer-wise optimization using trust ratios. Though prevailing, such methods are observed to still suffer from unstable and extreme trust ratios which degrades performance. In this paper, we propose a new variant of LAMB, called LAMBC, which employs trust ratio clipping to stabilize its magnitude and prevent extreme values. We conducted experiments on image classification tasks such as ImageNet and CIFAR-10 and our empirical results demonstrate promising improvements across different batch sizes.
Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?
Department of Computer Science, Purdue University, West Lafayette, IN, USA（计算机科学系，普渡大学，西拉法塞特，印第安纳州，美国）；School of Industrial Engineering, Purdue University, West Lafayette, IN, USA（工业工程学院，普渡大学，西拉法塞特，印第安纳州，美国）
University of Science and Technology of China（中国科学技术大学）；Stanford University（斯坦福大学）；Sun Yat-sen University（中山大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Research（微软研究院）
Differentially private deep learning has recently witnessed advances in computational efficiency and privacy-utility trade-off. We explore whether further improvements along the two axes are possible and provide affirmative answers leveraging two instantiations of \emph{group-wise clipping}. To reduce the compute time overhead of private learning, we show that \emph{per-layer clipping}, where the gradient of each neural network layer is clipped separately, allows clipping to be performed in conjunction with backpropagation in differentially private optimization. This results in private learnin…
In importance sampling (IS)-based reinforcement learning algorithms such as Proximal Policy Optimization (PPO), IS weights are typically clipped to avoid large variance in learning. However, policy update from clipped statistics induces large bias in tasks with high action dimensions, and bias from clipping makes it difficult to reuse old samples with large IS weights. In this paper, we consider PPO, a representative on-policy algorithm, and propose its improvement by dimension-wise IS weight clipping which separately clips the IS weight of each action dimension to avoid large bias and adaptiv…
University of Manchester（曼彻斯特大学）；University of Oxford（牛津大学）；Microsoft Research（微软研究院）
We present trust region bounds for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds even when the transition dynamics are non-stationary. This new analysis provides a theoretical understanding of the strong performance of two recent actor-critic methods for MARL, which both rely on independent ratios, i.e., computing probability ratios separately for each agent's policy. We show that, despite the non-stationarity that independent ratios cause, a monotonic improvement guarantee still arises as a result of enforcing the trust region constrai…
Mitigating Disparate Impact of Differentially Private Learning through Bounded Adaptive Clipping
Department of Computer Science, University of Helsinki（计算机科学系，赫尔辛基大学）；Department of Electrical and Computer Engineering, American University of Beirut（电气与计算机工程系，贝鲁特美国大学）
Privacy preserving machine learning algorithms are crucial for learning models over user data to protect sensitive information. Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed. At each step, these algorithms modify the gradients and add noise proportional to the sensitivity of the modified gradients. Under this framework, we propose AdaCliP, a theoretically motivated differentially private SGD algorithm that provably adds less noise compared to the previous methods, by using coordinate-wise adaptive …
Beihang University（北京航空航天大学）；SenseTime Research（商汤科技研究院）；University of Electronic Science and Technology of China（电子科技大学）；Peking University（北京大学）
Transformer architecture has become the fundamental element of the widespread natural language processing~(NLP) models. With the trends of large NLP models, the increasing memory and computation costs hinder their efficient deployment on resource-limited devices. Therefore, transformer quantization attracts wide research interest. Recent work recognizes that structured outliers are the critical bottleneck for quantization performance. However, their proposed methods increase the computation overhead and still leave the outliers there. To fundamentally address this problem, this paper delves in…
Very recently proximal policy optimization (PPO) algorithms have been proposed as first-order optimization methods for effective reinforcement learning. While PPO is inspired by the same learning theory that justifies trust region policy optimization (TRPO), PPO substantially simplifies algorithm design and improves data efficiency by performing multiple epochs of \emph{clipped policy optimization} from sampled data. Although clipping in PPO stands for an important new mechanism for efficient and reliable policy update, it may fail to adaptively improve learning performance in accordance with …
AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training
School of Artificial intelligence, JiLin University（人工智能学院，吉林大学）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, JiLin University（知识驱动人机智能工程研究中心，吉林大学）；International Center of Future Science, JiLin University（未来科学国际中心，吉林大学）
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Proximal policy optimization (PPO) is one of the most successful deep reinforcement-learning methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, its optimization behavior is still far from being fully understood. In this paper, we show that PPO could neither strictly restrict the likelihood ratio as it attempts to do nor enforce a well-defined trust region constraint, which means that it may still suffer from the risk of performance instability. To address this issue, we present an enhanced PPO method, named Truly PPO. Two critical improvements ar…
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across important tasks, such as attention models. The settings under which SGD performs poorly in comparison to adaptive methods are not well understood yet. In this paper, we provide empirical and theoretical evidence that a heavy-tailed distribution of the noise in stochastic gradients is one cause of SGD's poor performance. We provide the first tight upper and lower convergence bounds for adaptive gradient methods under …
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Proximal policy optimization (PPO) is one of the most popular deep reinforcement learning (RL) methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, as a model-free RL method, the success of PPO relies heavily on the effectiveness of its exploratory policy search. In this paper, we give an in-depth analysis on the exploration behavior of PPO, and show that PPO is prone to suffer from the risk of lack of exploration especially under the case of bad initialization, which may lead to the failure of training or being trapped in bad local optima. To addr…
TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers
Department of XXX, University of YYY, Location, Country；School of ZZZ, Institute of WWW, Location, Country；Department of Combinatorics；Optimization, University of Waterloo, Waterloo, Canada；Huawei Canadian Research Institute, Canada；McGill University \& Mila-Quebec AI Institue, Canada
Quantization is an essential step in the efficient deployment of deep learning models and as such is an increasingly popular research topic. An important practical aspect that is not addressed in the current literature is how to analyze and fix fail cases where the use of quantization results in excessive degradation. In this paper, we present a simple analytic framework that breaks down overall degradation to its per layer contributions. We analyze many common networks and observe that a layer's contribution is determined by both intrinsic (local) factors - the distribution of the layer's wei…
Department of Foundation Model, 2012 Labs, Huawei（华为基础模型部门，2012实验室）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（上海智能自主系统研究院，同济大学）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）
Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
Hong Kong University of Science and Technology（香港科技大学）；Mila - Québec AI Institute（魁北克AI研究院）；Université de Montréal（蒙特利尔大学）；Fudan University（复旦大学）；City University of Hong Kong（香港城市大学）
University of Bamberg（班贝格大学）；IBM Deutschland GmbH（IBM德国有限公司）；fortiss GmbH（福蒂斯有限公司）
Artificial Intelligence and Digital Twins play an integral role in driving innovation in the domain of intelligent driving. Long short-term memory (LSTM) is a leading driver in the field of lane change prediction for manoeuvre anticipation. However, the decision-making process of such models is complex and non-transparent, hence reducing the trustworthiness of the smart solution. This work presents an innovative approach and a technical implementation for explaining lane change predictions of layer normalized LSTMs using Layer-wise Relevance Propagation (LRP). The core implementation includes …
