Despite the ubiquitous use of stochastic optimization algorithms in machine learning, the precise impact of these algorithms and their dynamics on generalization performance in realistic non-convex settings is still poorly understood. While recent work has revealed connections between generalization and heavy-tailed behavior in stochastic optimization, this work mainly relied on continuous-time approximations; and a rigorous treatment for the original discrete-time iterations is yet to be performed. To bridge this gap, we present novel bounds linking generalization to the lower tail exponent of the transition kernel associated with the optimizer around a local minimum, in both discrete- and continuous-time settings. To achieve this, we first prove a data- and algorithm-dependent generalization bound in terms of the celebrated Fernique-Talagrand functional applied to the trajectory of the optimizer. Then, we specialize this result by exploiting the Markovian structure of stochastic optimizers, and derive bounds in terms of their (data-dependent) transition kernels. We support our theory with empirical results from a variety of neural networks, showing correlations between generalization error and lower tail exponents.
Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);School of Software Technology, Zhejiang University(浙江大学软件学院);School of Data Science, Fudan University(复旦大学数据科学学院)
Resource frugal optimizer for quantum machine learning
Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室);Leiden University(莱顿大学);Universidad Autónoma de Madrid(马德里自治大学);University of Warsaw(华沙大学);Normal Computing Corporation(诺默尔计算公司)
Reducing Bias in Deep Learning Optimization: The RSGDM Approach
Stevens Institute of Technology(史蒂文斯理工学院);University of California, Berkeley(加州大学伯克利分校);Chinese University of Hong Kong(香港中文大学);University of California, Irvine(加州大学欧文分校);Illinois Institute of Technology(伊利诺伊理工学院);Southern Methodist University(南卫理公会大学)