It is generally recognized that finite learning rate (LR), in contrast to infinitesimal LR, is important for good generalization in real-life deep nets. Most attempted explanations propose approximating finite-LR SGD with Ito Stochastic Differential Equations (SDEs), but formal justification for this approximation (e.g., (Li et al., 2019)) only applies to SGD with tiny LR. Experimental verification of the approximation appears computationally infeasible. The current paper clarifies the picture with the following contributions: (a) An efficient simulation algorithm SVAG that provably converges to the conventionally used Ito SDE approximation. (b) A theoretically motivated testable necessary condition for the SDE approximation and its most famous implication, the linear scaling rule (Goyal et al., 2017), to hold. (c) Experiments using this simulation to demonstrate that the previously proposed SDE approximation can meaningfully capture the training and generalization properties of common deep nets.
Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Inria（法国国家信息与自动化研究所）；Ecole Normale Supérieure（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）；Florida State University（佛罗里达州立大学）；Rutgers University（罗格斯大学）；Princeton University（普林斯顿大学）；CNRS（法国国家科学研究中心）
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
LTCI, Télécom ParisTech, Université Paris-Saclay（巴黎萨克雷大学巴黎高等电信学院LTCI实验室）；Institute of Physics, École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院物理研究所）；Rutgers Business School（罗格斯商学院）
Parameter Estimation in Stochastic Differential Equations via Wiener Chaos Expansion and Stochastic Gradient Descent
Universitat Autónoma de Barcelona, Departament de Matemàtiques（巴塞罗那自治大学数学系）；Facultad de Ciencias - Universidad Nacional Autónoma de México（墨西哥国立自治大学理学院）；Secihti - Centro Interdisciplinario de Ciencias Marinas. IPN.（国家理工学院海洋科学跨学科中心）
Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks
LTCI Télécom Paris（巴黎电信）；Institut Polytechnique de Paris（巴黎综合理工学院）；University of Oxford（牛津大学）；Intel Labs（英特尔研究院）；The Alan Turing Institute（艾伦·图灵研究所）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院）；University of Oxford（牛津大学）；Rutgers Business School（罗格斯商学院）；Facebook AI Research（脸书人工智能研究院）
Rényi Differential Privacy for Heavy-Tailed SDEs via Fractional Poincaré Inequalities
Inria, CNRS, Ecole Normale Supérieure PSL Research University（法国国家科学研究中心、巴黎高等师范研究所PSL研究大学）；Department of Management Science and Information Systems Rutgers Business School（罗格斯商学院管理科学与信息系统系）；School of Mathematics and Statistics Key Laboratory of Analytical Mathematics and Applications (Ministry of Education) Fujian Provincial Key Laboratory of Statistics and Artificial Intelligence Fujian Normal University（福建师范大学数学与统计学学院、福建省应用数学与分析重点实验室、福建省统计与人工智能重点实验室）；Faculty of Engineering and Natural Sciences Sabancı University（Sabancı大学工程与自然科学学院）；Department of Mathematics Florida State University（佛罗里达州立大学数学系）
Beyond Optimal Rates in Stochastic Optimization: Trajectory-Adaptive Stopping Rules
UC Berkeley（加州大学伯克利分校）；Inria（法国国家信息与自动化研究所）；École Normale Supérieure（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）；Department of Statistics, UC Berkeley（加州大学伯克利分校统计系）
Optimization-Induced Dynamics of Lipschitz Continuity in Neural Networks
University of Galway（Galway大学）；Irish National Centre for Research Training in AI (CRT-AI)（爱尔兰人工智能研究培训中心）；Université Laval（拉瓦尔大学）；Canada-CIFAR AI Chair（加拿大CIFAR人工智能 chair）；Mila - Québec AI Institute（魁北克人工智能研究所）；University of Toronto（多伦多大学）；MBZUAI, UAE（阿联酋MBZUAI）
On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach
University of Basel（巴塞尔大学）；University of Oslo（奥斯陆大学）；MBZUAI（马克斯·普朗克智能系统研究所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Tübingen AI Center（图宾根人工智能中心）
