Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network parameters into blocks, and use a blockwise adaptive stepsize. Intuitively, blockwise adaptivity is less aggressive than adaptivity to individual coordinates, and can have a better balance between adaptivity and generalization. We show theoretically that the proposed blockwise adaptive gradient descent has comparable convergence rate as its counterpart with coordinate-wise adaptive stepsize, but is faster up to some constant. We also study its uniform stability and show that blockwise adaptivity can lead to lower generalization error than coordinate-wise adaptivity. Experimental results show that blockwise adaptive gradient descent converges faster and improves generalization performance over Nesterov's accelerated gradient and Adam.
Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization
Department of Computer Science, Purdue University, West Lafayette, USA(计算机科学系,普渡大学,西拉法叶,美国);Edwardson School of Industrial Engineering, Purdue University, West Lafayette, USA(工业工程学院,普渡大学,西拉法叶,美国)
Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search)
University of Alberta(阿尔伯塔大学);Mila, McGill University(米拉研究所,麦吉尔大学);University of British Columbia(不列颠哥伦比亚大学);Cornell University(康奈尔大学);Mila, Université de Montréal(米拉研究所,蒙特利尔大学)
Conda: Column-Normalized Adam for Training Large Language Models Faster
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,智能科学与技术学院,北京大学);Singapore Management University(新加坡国立大学);Nankai University(南开大学);Beijing Normal University(北京师范大学);ByteDance Seed(字节跳动种子);Beijing University of Posts and Telecommunications(北京邮电大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学);Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China(黄埔实验室(广州),广东,中国)