We consider unconstrained minimization of smooth convex functions. We propose a novel variational perspective using forced Euler-Lagrange equation that allows for studying high-resolution ODEs. Through this, we obtain a faster convergence rate for gradient norm minimization using Nesterov's accelerated gradient method. Additionally, we show that Nesterov's method can be interpreted as a rate-matching discretization of an appropriately chosen high-resolution ODE. Finally, using the results from the new variational perspective, we propose a stochastic method for noisy gradients. Several numerical experiments compare and illustrate our stochastic algorithm with state of the art methods.
Understanding the Acceleration Phenomenon via High-Resolution Differential Equations
Florida International University（佛罗里达国际大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）；University of Pennsylvania（宾夕法尼亚大学）
Provably Correct Learning Algorithms in the Presence of Time-Varying Features Using a Variational Perspective
Massachusetts Institute of Technology（麻省理工学院）；Brigham and Women’s Hospital and Harvard Medical School（布里奇沃特医院和哈佛医学院）；Air Force Research Laboratory（空军研究实验室）
A Dynamical Systems Perspective on the Analysis of Neural Networks
Freie Universität Berlin（柏林自由大学）；Universiteit van Amsterdam（阿姆斯特丹大学）；Technical University of Munich（慕尼黑技术大学）；Munich Data Science Institute（慕尼黑数据科学研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Layerwise goal-oriented adaptivity for neural ODEs: an optimal control perspective
Institute for Mathematics, Humboldt-Universität zu Berlin, Berlin, Germany（柏林洪堡大学数学研究所）；Weierstrass Institute for Applied Analysis and Stochastics, Berlin, Germany（魏泽拉施应用分析与随机学研究所）；Mathematical Institute, Universität Koblenz, Koblenz, Germany（科布伦茨大学数学研究所）
A Variational Inequality Perspective on Generative Adversarial Networks
Mila & DIRO, University of Montreal（蒙特利尔大学 Mila 与 DIRO）；Facebook Artificial Intelligence Research（Facebook 人工智能研究院）；Canada CIFAR AI Chair（加拿大 CIFAR 人工智能教席）
