arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.09045math.OCcs.CCcs.DC
收藏

寻找驻点的自适应复杂度

The Adaptive Complexity of Finding a Stationary Point

Huanjian Zhou, Andi Han, Akiko Takeda, Masashi Sugiyama

AI总结:

本文研究并行非凸优化中寻找驻点的自适应复杂度,分别给出高维迭代轮数下界与常数维查询数下界,证明梯度下降、高阶自适应正则化及梯度流捕获等方法自适应最优。

摘要:

在机器学习等大规模应用中,人们希望设计具有高度并行化能力的非凸优化算法。本文研究寻找驻点的自适应复杂度,即在每轮并行执行多项式多次查询的条件下,达到驻点所需的最少顺序轮数。对于高维情形,即 $d=\widetildeΩ(\varepsilon^{-(2+2p)/p})$,我们证明:对于任何(可能随机化的)算法,都存在一个具有 Lipschitz 连续 $p$ 阶导数的函数,使该算法至少需要 $\varepsilon^{-(p+1)/p}$ 次迭代才能找到一个 $\varepsilon$-驻点。我们的下界是紧的,并表明即使每次迭代使用 $\mathrm{poly}(d)$ 次查询,也没有算法能获得优于每轮一次查询算法可达到的收敛率。换言之,梯度下降、三次正则化 Newton 法以及 $p$ 阶自适应正则化方法都是自适应最优的。我们的证明依赖于新颖分析:利用带随机划分的链状结构刻画困难性势函数的输出。对于常数维情形,即 $d=Θ(1)$,我们提出一种连接网格搜索与梯度流捕获的算法,可在常数次迭代内找到近似驻点。其渐近紧性由关于每轮迭代所需查询数的一个新下界验证。我们证明存在一个光滑函数,使任何以 $Θ(\log(1/\varepsilon))$ 轮运行的算法,每轮至少需要 $\widetildeΩ((1/\varepsilon)^{(d-1)/2})$ 次查询。该下界在相差一个对数因子的意义下是紧的,并意味着梯度流捕获具有自适应最优性。

英文摘要:

In large-scale applications, such as machine learning, it is desirable to design non-convex optimization algorithms with a high degree of parallelization. In this work, we study the adaptive complexity of finding a stationary point, which is the minimal number of sequential rounds required to achieve stationarity given polynomially many queries executed in parallel at each round. For the high-dimensional case, i.e., $d = \widetildeΩ(\varepsilon^{-(2 + 2p)/p})$, we show that for any (potentially randomized) algorithm, there exists a function with Lipschitz $p$-th order derivatives such that the algorithm requires at least $\varepsilon^{-(p+1)/p}$ iterations to find an $\varepsilon$-stationary point. Our lower bounds are tight and show that even with $\mathrm{poly}(d)$ queries per iteration, no algorithm has better convergence rate than those achievable with one-query-per-round algorithms. In other words, gradient descent, the cubic-regularized Newton's method, and the $p$-th order adaptive regularization method are adaptively optimal. Our proof relies upon novel analysis with the characterization of the output for the hardness potentials based on a chain-like structure with random partition. For the constant-dimensional case, i.e., $d = Θ(1)$, we propose an algorithm that bridges grid search and gradient flow trapping, finding an approximate stationary point in constant iterations. Its asymptotic tightness is verified by a new lower bound on the required queries per iteration. We show there exists a smooth function such that any algorithm running with $Θ(\log (1/\varepsilon))$ rounds requires at least $\widetildeΩ((1/\varepsilon)^{(d-1)/2})$ queries per round. This lower bound is tight up to a logarithmic factor, and implies that the gradient flow trapping is adaptively optimal.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑