Heriot-Watt University（赫瑞-瓦特大学）；University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）
Mateusz B. Majka, Marc Sabate-Vidales, Łukasz Szpruch
备注：
51 pages, 8 figures. Revised version: an improved introduction, a completely new numerical section including experiments in non-convex settings, a new appendix discussing the dependence of the variance of SGLD on the mini-batch size
Stochastic Gradient Algorithms (SGAs) are ubiquitous in computational statistics, machine learning and optimisation. Recent years have brought an influx of interest in SGAs, and the non-asymptotic analysis of their bias is by now well-developed. However, relatively little is known about the optimal choice of the random approximation (e.g mini-batching) of the gradient in SGAs as this relies on the analysis of the variance and is problem specific. While there have been numerous attempts to reduce the variance of SGAs, these typically exploit a particular structure of the sampled distribution by requiring a priori knowledge of its density's mode. It is thus unclear how to adapt such algorithms to non-log-concave settings. In this paper, we construct a Multi-index Antithetic Stochastic Gradient Algorithm (MASGA) whose implementation is independent of the structure of the target measure and which achieves performance on par with Monte Carlo estimators that have access to unbiased samples from the distribution of interest. In other words, MASGA is an optimal estimator from the mean square error-computational cost perspective within the class of Monte Carlo estimators. We prove this fact rigorously for log-concave settings and verify it numerically for some examples where the log-concavity assumption is not satisfied.
Gradient descent dynamics for deep equilibrium models
Carnegie Mellon University, Department of Statistics（卡内基梅隆大学统计系）；Carnegie Mellon University, Department of Statistics and Machine Learning（卡内基梅隆大学统计与机器学习系）
On Learning Gaussian Multi-index Models with Gradient Flow
Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；Center for Data Science, New York University（纽约大学数据科学中心）；Center for Computational Mathematics, Flatiron Institute（熨斗研究所计算数学中心）
Average Gradient Outer Product in kernel regression provably recovers the central subspace for multi-index models
Department of Mathematics, University of Washington, Seattle, WA 98195（华盛顿大学数学系，华盛顿州西雅图98195）；Wharton Department of Statistics and Data Science, University of Pennsylvania, Philadelphia, PA 19104, USA（宾夕法尼亚大学沃顿统计与数据科学系，美国费城19104）；Department of Mathematics, U. Washington, Seattle, WA 98195（华盛顿大学数学系，华盛顿州西雅图98195）；Department of Electrical & Computer Engineering, University of Washington, Seattle, WA 98195, and Amazon, Inc（华盛顿大学电气与计算机工程系，华盛顿州西雅图98195，亚马逊公司）
Institute of Science and Technology Austria（奥地利科学与技术研究所）；Sung Kyun Kwan University（顺天妇女大学）；New York University and Flatiron Institute（纽约大学和Flatiron研究所）；University of Southern California（南加州大学）
Emergence of heavy tails in homogenized stochastic gradient descent
School of Mathematics and Statistics, Northwestern Polytechnical University（西北工业大学数学与统计学院）；TU Dresden（德累斯顿工业大学）；ScaDS.ai Center for scalable data analytics and artificial intelligence（ScaDS.ai可扩展数据分析与人工智能中心）
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
LTCI, Télécom ParisTech, Université Paris-Saclay（巴黎萨克雷大学巴黎高等电信学院LTCI实验室）；Institute of Physics, École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院物理研究所）；Rutgers Business School（罗格斯商学院）
Courant Institute of Mathematical Sciences（数学科学学院）；Center for Data Science, New York University（数据科学中心，纽约大学）；Department of Computer Science（计算机科学系）；Data Science Institute, Columbia University（数据科学学院，哥伦比亚大学）
