Recent research on online Gradient Balancing (GraB) has revealed that there exist permutation-based example orderings for SGD that are guaranteed to outperform random reshuffling (RR). Whereas RR arbitrarily permutes training examples, GraB leverages stale gradients from prior epochs to order examples -- achieving a provably faster convergence rate than RR. However, GraB is limited by design: while it demonstrates an impressive ability to scale-up training on centralized data, it does not naturally extend to modern distributed ML workloads. We therefore propose Coordinated Distributed GraB (CD-GraB), which uses insights from prior work on kernel thinning to translate the benefits of provably faster permutation-based example ordering to distributed settings. With negligible overhead, CD-GraB exhibits a linear speedup in convergence rate over centralized GraB and outperforms distributed RR on a variety of benchmark tasks.
Provably Correct Learning Algorithms in the Presence of Time-Varying Features Using a Variational Perspective
Massachusetts Institute of Technology（麻省理工学院）；Brigham and Women’s Hospital and Harvard Medical School（布里奇沃特医院和哈佛医学院）；Air Force Research Laboratory（空军研究实验室）
Katyusha X: Practical Momentum Method for Stochastic Sum-of-Nonconvex Optimization
Microsoft Research AI（微软研究院AI）
最小化非凸函数之和的问题（即，凸函数（即非凸函数的平均值）在机器学习中变得越来越重要，并且是PCA、SVD、正则化牛顿法、加速非凸优化等的核心机制。我们展示了如何可证明地获得一个加速的随机算法最小化的总和的非凸函数，由$\textit{adding one additional line}$著名的SVRG方法。这条线对应于动量，并显示了如何直接将动量应用于非凸函数之和的有限和随机最小化。作为一个副作用，我们的方法享有线性并行加速使用小批量。
Accelerated Learning with Robustness to Adversarial Regressors
Massachusetts Institute of Technology（麻省理工学院）；Air Force Research Laboratory（空军研究实验室）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Fast quantum state reconstruction via accelerated non-convex programming
Rice University（莱斯大学）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）；University of Southern California（南加州大学）；Information Sciences Institute（信息科学研究所）
Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Vietnam Academy of Science and Technology（越南科学技术研究院）；Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）深圳大数据研究院）
CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimization
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI)（SDAIA-KAUST数据科学与人工智能卓越中心）；Brno University of Technology（布尔诺理工大学）；Kempelen Institute of Intelligent Technologies (KInIT)（肯佩伦智能技术研究所）
Near-Optimal Convergence of Accelerated Gradient Methods under Generalized and $(L_0, L_1)$-Smoothness
Applied AI Institute, Moscow, Russia（应用人工智能研究所，莫斯科，俄罗斯）
本文研究了在满足最近提出的$\ell$-光滑性条件的凸优化问题中的一阶方法。虽然加速梯度下降法（AGD）在$L$-光滑性下能达到最优复杂度$O(\sqrt{L} R / \sqrt{\varepsilon})$，但现有方法在$\ell$-光滑性下的扩展要么引入额外的初始梯度依赖，要么有指数因子$ L_1 R $，或者需要昂贵的辅助子程序。本文解决了这一开放问题，通过新的Lyapunov函数和设计新的算法，实现了$O(\sqrt{\ell(0)} R / \sqrt{\varepsilon})$的oracle复杂度，对于小$\varepsilon$和几乎任何$\ell$。例如，在$(L_0, L_1)$-光滑性下，我们的界$O(\sqrt{L_0} R / \sqrt{\varepsilon})$在小$\varepsilon$范围内被证明是最佳的，并去除了先前加速算法中所有非常数的乘法因子。
Provably Accelerated Decentralized Gradient Method Over Unbalanced Directed Graphs
Shanghai Center for Mathematical Sciences, Fudan University（复旦大学上海数学中心）；School of Mathematical Sciences, Fudan University（复旦大学数学科学学院）；Shanghai Key Laboratory for Contemporary Applied Mathematics（上海现代应用数学重点实验室）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳）数据科学学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI)（沙特数据与人工智能局-阿卜杜拉国王科技大学数据科学与人工智能卓越中心）
Faster Margin Maximization Rates for Generic and Adversarially Robust Optimization Methods
College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）；Google Research（谷歌研究院）；School of Electrical and Computer Engineering, Georgia Institute of Technology（佐治亚理工学院电气与计算机工程学院）；School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院工业与系统工程学院）
