We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close relationship to the Newton method, and the second one the Fisher type as its preconditioner is closely related to the inverse of Fisher information matrix. Both preconditioners can be derived from one framework, and efficiently estimated on any matrix Lie groups designated by the user using natural or relative gradient descent minimizing certain preconditioner estimation criteria. Many existing preconditioners and methods, e.g., RMSProp, Adam, KFAC, equilibrated SGD, batch normalization, etc., are special cases of or closely related to either the Newton type or the Fisher type ones. Experimental results on relatively large scale machine learning problems are reported for performance study.
Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability
Department of Mathematics, Emory University（埃默里大学数学系）；Department of Mathematics, University of Minnesota Twin Cities（明尼苏达大学双城分校数学系）；Department of Computer Science, University of Minnesota Twin Cities（明尼苏达大学双城分校计算机科学系）；Department of Mathematics, University of Kentucky（肯塔基大学数学系）
High-dimensional limit theorems for SGD: Momentum and Adaptive Step-sizes
Department of Statistics and Actuarial Science, University of Waterloo, Canada（统计与精算科学系，滑铁卢大学，加拿大）；Cheriton School of Computer Science, University of Waterloo, Canada（切尔顿计算机科学学院，滑铁卢大学，加拿大）
Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noise
Department of Mathematics, University of California, Berkeley, CA, USA（加州大学伯克利分校数学系）；Department of EECS, University of California, Berkeley, CA, USA（加州大学伯克利分校电子工程与计算机科学系）；Department of IEOR, University of California, Berkeley, CA, USA（加州大学伯克利分校工业工程与运营研究系）
Statistical Guarantees for High-Dimensional Stochastic Gradient Descent
Department of Statistics University of Chicago（统计学系芝加哥大学）；Department of Mathematics University of California, San Diego（数学系加州大学圣地亚哥分校）；Department of Applied Mathematics University of Twente（应用数学系埃因霍温大学）
Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD
Paris Research Center, Huawei Technologies France（华为技术法国巴黎研发中心）；Boğaziçi University（博加齐大学）；Université Gustave Eiffel（古斯塔夫·埃菲尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Ecole Normale Supérieure（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）
On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD
Sch. of Computer Science and Zhiyuan College, Shanghai Jiao Tong University（上海交通大学计算机科学学院和智远学院）；Peking University（北京大学）；University of Sydney（悉尼大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；The Institute of Statistical Mathematics（统计数学研究所）；The University of Tokyo（东京大学）
LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎高等电信学院LTCI实验室）；University of British Columbia（英属哥伦比亚大学）；Amii（阿尔伯塔机器智能研究所）；Inria - PSL Research University（法国国家信息与自动化研究所-巴黎文理研究大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Gradient is All You Need? How Consensus-Based Optimization can be Interpreted as a Stochastic Relaxation of Gradient Descent
University of Oxford, Mathematical Institute（牛津大学数学研究所）；Deeptech Consulting（德普科技咨询）；ETH Zurich, Department of Mathematics（苏黎世联邦理工学院数学系）；Technical University of Munich, School of Computation, Information and Technology, Department of Mathematics（慕尼黑技术大学计算、信息与技术学院数学系）；Munich Center for Machine Learning Munich Data Science Institute（慕尼黑机器学习中心慕尼黑数据科学研究所）
Learning Operators by Regularized Stochastic Gradient Descent with Operator-valued Kernels
School of Mathematical Sciences, Fudan University（复旦大学数学科学学院）；Fudan University（复旦大学）；Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University（复旦大学当代应用数学重点实验室）
Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)（人工智能基础研究实验室（BRAIn Lab））；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（人工智能 Mohamed bin Zayed 大学）；Innopolis University（Innopolis 大学）
