arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.10526math.OCcs.DScs.LGmath.STstat.MLstat.TH
收藏

具有与层无关收敛速度的随机多层复合Optimization算法

Stochastic Multi-level Composition Optimization Algorithms with Level-Independent Convergence Rates

University of California, Davis(加州大学戴维斯分校);University of Waterloo(滑铁卢大学)

Krishnakumar Balasubramanian, Saeed Ghadimi, Anthony Nguyen

摘要:

本文研究光滑随机多层复合最优化问题,其中目标函数是$T$函数的嵌套复合。我们假设通过随机一阶预言机访问函数及其梯度的噪声评估。对于这类问题,我们提出了两个算法使用移动平均随机估计,并分析其收敛到一个$ε$ -平稳点的问题。我们表明,第一个算法,这是一个推广的\cite{GhaRuswan20}的$T$级别的情况下,可以实现一个样本的复杂性$\mathcal{O}(1/ε^6)$,通过使用小批量的样本在每次迭代。通过使用函数值的线性化随机估计来修改该算法,我们将样本复杂度提高到$\mathcal{O}(1/ε^4)$。{ \color{black}此修改不仅消除了在每次迭代中具有小批量样本的要求,而且使算法无参数且易于实现}。据我们所知,这是第一次,这样的在线算法设计的(无)约束的多级设置,获得相同的样本复杂性的光滑单级设置,在标准假设(无偏性和有界性的二阶矩)的随机一阶预言。

英文摘要:

In this paper, we study smooth stochastic multi-level composition optimization problems, where the objective function is a nested composition of $T$ functions. We assume access to noisy evaluations of the functions and their gradients, through a stochastic first-order oracle. For solving this class of problems, we propose two algorithms using moving-average stochastic estimates, and analyze their convergence to an $ε$-stationary point of the problem. We show that the first algorithm, which is a generalization of \cite{GhaRuswan20} to the $T$ level case, can achieve a sample complexity of $\mathcal{O}(1/ε^6)$ by using mini-batches of samples in each iteration. By modifying this algorithm using linearized stochastic estimates of the function values, we improve the sample complexity to $\mathcal{O}(1/ε^4)$. {\color{black}This modification not only removes the requirement of having a mini-batch of samples in each iteration, but also makes the algorithm parameter-free and easy to implement}. To the best of our knowledge, this is the first time that such an online algorithm designed for the (un)constrained multi-level setting, obtains the same sample complexity of the smooth single-level setting, under standard assumptions (unbiasedness and boundedness of the second moments) on the stochastic first-order oracle.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑