The study of first-order optimization algorithms (FOA) typically starts with assumptions on the objective functions, most commonly smoothness and strong convexity. These metrics are used to tune the hyperparameters of FOA. We introduce a class of perturbations quantified via a new norm, called *-norm. We show that adding a small perturbation to the objective function has an equivalently small impact on the behavior of any FOA, which suggests that it should have a minor impact on the tuning of the algorithm. However, we show that smoothness and strong convexity can be heavily impacted by arbitrarily small perturbations, leading to excessively conservative tunings and convergence issues. In view of these observations, we propose a notion of continuity of the metrics, which is essential for a robust tuning strategy. Since smoothness and strong convexity are not continuous, we propose a comprehensive study of existing alternative metrics which we prove to be continuous. We describe their mutual relations and provide their guaranteed convergence rates for the Gradient Descent algorithm accordingly tuned. Finally we discuss how our work impacts the theoretical understanding of FOA and their performances.
Distributionally Time-Varying Online Stochastic Optimization under Polyak-Łojasiewicz Condition with Application in Conditional Value-at-Risk Statistical Learning
The Australian National University（澳大利亚国立大学）；The University of Melbourne（墨尔本大学）
KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey
University of Waterloo（滑铁卢大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Statistics and Actuarial Science（统计与精算科学系）；David R. Cheriton School of Computer Science（大卫·R·谢里顿计算机科学学院）；Centre for Pattern Analysis and Machine Intelligence（模式分析与机器智能中心）；Machine Learning Laboratory（机器学习实验室）；Data Analytics Laboratory（数据分析实验室）
SC Johnson College of Business, Cornell University（康奈尔大学约翰逊管理学院）；Department of Industrial Engineering and Operations Research, UC Berkeley（加州大学伯克利分校工业工程与运筹学系）；Faculty of Engineering & Faculty of Business and Economics, University of Hong Kong（香港大学工程学院及商学院）
An Accelerated Variance-Reduced Conditional Gradient Sliding Algorithm for First-order and Zeroth-order Optimization
Nanjing University of Information Science & Technology（南京信息工程大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；JD Finance America Corporation（京东金融美国公司）；University of Pittsburgh（匹兹堡大学）
Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimization
Nanyang Technological University（南洋理工大学）；Imperial College London（伦敦帝国理工学院）；The Alan Turing Institute（艾伦·图灵研究所）；The University of Warwick（华威大学）；The University of Edinburgh（爱丁堡大学）；National Technical University of Athens（雅典国立技术大学）
