In this paper, a new theory is developed for first-order stochastic convex optimization, showing that the global convergence rate is sufficiently quantified by a local growth rate of the objective function in a neighborhood of the optimal solutions. In particular, if the objective function $F(\mathbf w)$ in the $ε$-sublevel set grows as fast as $\|\mathbf w - \mathbf w_*\|_2^{1/θ}$, where $\mathbf w_*$ represents the closest optimal solution to $\mathbf w$ and $θ\in(0,1]$ quantifies the local growth rate, the iteration complexity of first-order stochastic optimization for achieving an $ε$-optimal solution can be $\widetilde O(1/ε^{2(1-θ)})$, which is optimal at most up to a logarithmic factor. To achieve the faster global convergence, we develop two different accelerated stochastic subgradient methods by iteratively solving the original problem approximately in a local region around a historical solution with the size of the local region gradually decreasing as the solution approaches the optimal set. Besides the theoretical improvements, this work also includes new contributions towards making the proposed algorithms practical: (i) we present practical variants of accelerated stochastic subgradient methods that can run without the knowledge of multiplicative growth constant and even the growth rate $θ$; (ii) we consider a broad family of problems in machine learning to demonstrate that the proposed algorithms enjoy faster convergence than traditional stochastic subgradient method. We also characterize the complexity of the proposed algorithms for ensuring the gradient is small without the smoothness assumption.
Accelerated stochastic first-order method for convex optimization under heavy-tailed noise
Department of Mathematics, Linköping University, Sweden(林雪平大学数学系);Department of Industrial and Systems Engineering, University of Minnesota, USA(明尼苏达大学工业与系统工程系)
Linear convergence of forward-backward accelerated algorithms without knowledge of the modulus of strong convexity
Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);School of Mathematical Sciences, University of Chinese Academy of Sciences(中国科学院大学数学科学学院)
A Unified Analysis on the Subgradient Upper Bounds for the Subgradient Methods Minimizing Composite Nonconvex, Nonsmooth and Non-Lipschitz Functions
Antai College of Economics and Management, Shanghai Jiao Tong University(上海交通大学安泰经济管理学院);School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);Institute of Translational Medicine and National Center for Translational Medicine, Shanghai Jiao Tong University(上海交通大学转化医学研究院和转化医学中心);Department of Industrial and Systems Engineering, University of Minnesota(明尼苏达大学工业与系统工程系)
Leveraging Legacy Data to Accelerate Materials Design via Preference Learning
Graduate School of Frontier Sciences, The University of Tokyo(东京大学前沿科学研究生院);Research and Services Division of Materials Data and Integrated System, National Institute for Materials Science(国立材料科学研究所材料数据与集成系统研究及服务部);State Key Laboratory of Structural Chemistry, Fujian Institute of Research on the Structure of Matter, Chinese Academy of Sciences (CAS)(中国科学院福建物质结构研究所结构化学国家重点实验室);International Center for Materials Nanoarchitectonics (WPI-MANA), National Institute for Materials Science(国立材料科学研究所国际材料纳米构造学中心(WPI-MANA));Center for Advanced Intelligence Project, RIKEN(理化学研究所先进智能项目中心)
The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization
Command and Control Engineering College, Army Engineering University of PLA(中国人民解放军陆军工程大学指挥与控制工程学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Army Academy Of Artillery and Air Defense(陆军防空兵学院)
Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);University of Chinese Academy of Sciences(中国科学院大学)
在本文中,我们重新审视类迭代收缩阈值算法(ISTA)解决线性逆问题的稀疏表示,这出现在信号和图像处理。对图像去模糊的数值实验表明,该算法在坐标尺度上的收敛行为是线性的,而不是对数的,近似于平坦的。通过细致的观察,我们发现前面的假设是光滑的部分是凸的,削弱了最小二乘模型。具体地说,假设平滑部分是强凸的对于最小二乘模型是更合理的,即使图像矩阵可能是病态的。此外,我们改进了[Li et al.,2022年]。基于这个关键不等式,我们将复合优化问题的线性收敛推广到目标值和平方次梯度范数下。同时,我们设置了一个简单的病态矩阵来代替原来的模糊矩阵,便于计算奇异值。新的数值实验表明,Nesterov的加速梯度下降(NAG)的强凸函数的近端推广比ISTA具有更快的线性收敛速率。基于更紧的枢轴不等式,我们还推广了更快的线性收敛速率的复合优化,在目标值和平方近次梯度范数,利用构造良好的李雅普诺夫函数与轻微的修改和相空间表示的基础上的高分辨率微分方程框架的隐速度计划。
Leveraging Operator Learning to Accelerate Convergence of the Preconditioned Conjugate Gradient Method
Toulouse-INP (ENSEEIHT)/IRIT(图卢兹INP(ENSEEIHT)/IRIT);Artificial and Natural Intelligence Toulouse Institute (ANITI)(人工智能与自然智能图卢兹研究所(ANITI));Division of Applied Mathematics, Brown University(应用数学系,布朗大学)