加速非精确超梯度下降求解双层优化

中文摘要 AI 辅助

我们提出了一种求解一般非凸-强凸双层优化问题的方法。我们的方法——重启加速超梯度下降（RAHGD）方法——能以 $\tilde{\mathcal{O}}(κ^{3.25}ε^{-1.75})$ 的预言机复杂度找到目标函数的 $ε$-一阶驻点，其中 $κ$ 是下层目标的条件数，$ε$ 是期望精度。我们还提出了 RAHGD 的扰动变体，用于在相同阶的预言机复杂度内找到 $\big(ε,\mathcal{O}(κ^{2.5}\sqrtε\\,)\big)$-二阶驻点。我们的结果实现了双层优化中寻找驻点的已知最佳理论保证，并改进了非凸-强凹极小极大优化问题中寻找二阶驻点的现有复杂度上界，设定了新的最优基准。我们进行了实证研究以验证本文的理论结果。

英文摘要

We present a method for solving general nonconvex-strongly-convex bilevel optimization problems. Our method -- the \emph{Restarted Accelerated HyperGradient Descent} (\texttt{RAHGD}) method -- finds an $ε$-first-order stationary point of the objective with $\tilde{\mathcal{O}}(κ^{3.25}ε^{-1.75})$ oracle complexity, where $κ$ is the condition number of the lower-level objective and $ε$ is the desired accuracy. We also propose a perturbed variant of \texttt{RAHGD} for finding an $\big(ε,\mathcal{O}(κ^{2.5}\sqrtε\,)\big)$-second-order stationary point within the same order of oracle complexity. Our results achieve the best-known theoretical guarantees for finding stationary points in bilevel optimization and also improve upon the existing upper complexity bound for finding second-order stationary points in nonconvex-strongly-concave minimax optimization problems, setting a new state-of-the-art benchmark. Empirical studies are conducted to validate the theoretical results in this paper.

发表机构

• Fudan University（复旦大学）
• University of California, Berkeley（加州大学伯克利分校）

机构由 AI 辅助整理，请以论文原文为准。