arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.07530cs.LGmath.OC
收藏

随机多层组合优化的优化算法

Optimal Algorithms for Stochastic Multi-Level Compositional Optimization

Nanjing University(南京大学);The University of Iowa(爱荷华大学)

Wei Jiang, Bokun Wang, Yibo Wang, Lijun Zhang, Tianbao Yang

摘要:

本文研究了随机多层组合优化问题,其中目标函数是多个光滑但可能非凸的函数的组合。现有的解决这个问题的方法要么遭受次优的样本复杂性或需要一个巨大的批量大小。为了解决这些限制,我们提出了一种随机多水平方差减少方法(SMVR),它实现了最佳的样本复杂度的$\mathcal{O}\left(1 / ε^{3}\right)$找到一个$ε$ -平稳点的非凸目标。此外,当目标函数满足凸性或Polyak-Projojasiewicz(PL)条件时,我们提出了SMVR的一个阶段式变体,并将样本复杂度提高到对于凸函数的$\mathcal{O}\left(1 / ε^{2}\right)$或对于满足$μ$ -PL条件的非凸函数的$\mathcal{O}\left(1 /\left(με\right)\right)$.后者的结果意味着相同的复杂性$μ$ -强凸函数。为了利用自适应学习率,我们还开发了自适应SMVR,它实现了相同的复杂性,但在实践中收敛得更快。我们所有的复杂度不仅在$ε$方面,而且在$μ$(对于PL或强凸函数)方面都匹配下界,而无需在每次迭代中使用大批量。

英文摘要:

In this paper, we investigate the problem of stochastic multi-level compositional optimization, where the objective function is a composition of multiple smooth but possibly non-convex functions. Existing methods for solving this problem either suffer from sub-optimal sample complexities or need a huge batch size. To address these limitations, we propose a Stochastic Multi-level Variance Reduction method (SMVR), which achieves the optimal sample complexity of $\mathcal{O}\left(1 / ε^{3}\right)$ to find an $ε$-stationary point for non-convex objectives. Furthermore, when the objective function satisfies the convexity or Polyak-Łojasiewicz (PL) condition, we propose a stage-wise variant of SMVR and improve the sample complexity to $\mathcal{O}\left(1 / ε^{2}\right)$ for convex functions or $\mathcal{O}\left(1 /\left(με\right)\right)$ for non-convex functions satisfying the $μ$-PL condition. The latter result implies the same complexity for $μ$-strongly convex functions. To make use of adaptive learning rates, we also develop Adaptive SMVR, which achieves the same complexities but converges faster in practice. All our complexities match the lower bounds not only in terms of $ε$ but also in terms of $μ$ (for PL or strongly convex functions), without using a large batch size in each iteration.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑