arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01959math.OCcs.LG
收藏

更弱假设下随机优化的自适应方差缩减

Adaptive Variance Reduction for Stochastic Optimization under Weaker Assumptions

National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)

Wei Jiang, Sifan Yang, Yibo Wang, Lijun Zhang

AI总结:

本文针对STORM自适应方差缩减方法在强假设和额外对数项方面的局限,提出新的自适应方法,在更弱假设下实现非凸优化最优收敛速率,并扩展至复合优化和有限和问题。

摘要:

本文探讨了基于STORM技术的随机优化自适应方差缩减方法。现有的STORM自适应扩展要么依赖于有界梯度和有界函数值等强假设,要么在收敛速率中额外引入一个$\mathcal{O}(\log T)$项。为了解决这些局限,我们提出一种新颖的自适应STORM方法,借助我们新设计的学习率策略,对非凸函数实现了$\mathcal{O}(T^{-1/3})$的最优收敛速率。与现有方法相比,我们的方法所需假设更弱,并且在不引入额外$\mathcal{O}(\log T)$项的情况下达到了最优收敛速率。我们还将所提出的技术扩展到随机复合优化中,获得了相同的$\mathcal{O}(T^{-1/3})$最优速率。此外,我们研究了非凸有限和问题,并开发了另一种创新的自适应方差缩减方法,实现了$\mathcal{O}(n^{1/4} T^{-1/2})$的最优收敛速率,其中$n$表示分量函数的数量。在多种任务上的数值实验验证了我们方法的有效性。

英文摘要:

This paper explores adaptive variance reduction methods for stochastic optimization based on the STORM technique. Existing adaptive extensions of STORM rely on strong assumptions like bounded gradients and bounded function values, or suffer an additional $\mathcal{O}(\log T)$ term in the convergence rate. To address these limitations, we introduce a novel adaptive STORM method that achieves an optimal convergence rate of $\mathcal{O}(T^{-1/3})$ for non-convex functions with our newly designed learning rate strategy. Compared with existing approaches, our method requires weaker assumptions and attains the optimal convergence rate without the additional $\mathcal{O}(\log T)$ term. We also extend the proposed technique to stochastic compositional optimization, obtaining the same optimal rate of $\mathcal{O}(T^{-1/3})$. Furthermore, we investigate the non-convex finite-sum problem and develop another innovative adaptive variance reduction method that achieves an optimal convergence rate of $\mathcal{O}(n^{1/4} T^{-1/2} )$, where $n$ represents the number of component functions. Numerical experiments across various tasks validate the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑