发表机构
The Pennsylvania State University; The Chinese University of Hong Kong, Shenzhen（宾夕法尼亚州立大学; 香港中文大学（深圳））
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究针对非凸-强凸双层优化，提出单循环一阶算法SGHA及随机版本Stoc-SGHA，实现了更优的预言机复杂度，匹配相关下界的ε依赖关系。
AI 中文摘要
在本研究中，我们仅使用一阶预言机，探究寻找非凸-强凸（NC-SC）双层优化的ε-驻点的预言机复杂度。现有能达到最优复杂度保证的方法通常依赖基于惩罚的双循环流程。我们提出一种基于约束重表述的新型单循环算法，其中下层驻点被作为约束施加。具体而言，我们通过引入二次正则项并将对偶变量限制在有界域内，构造了正则化拉格朗日函数，随后应用平滑梯度下降上升（Smoothed Gradient Descent Ascent）[Zhang等人，2020]，其中海森向量积通过梯度的有限差分近似。我们将得到的确定性算法和随机算法分别命名为SGHA和Stoc-SGHA。在确定性场景下，SGHA达到的预言机复杂度为O(κ̄ᵧ⁵ε⁻²)，其中κ̄ᵧ表示相关条件数。在随机场景下，对于任意ρ∈(0,1)，Stoc-SGHA以至少1-ρ的概率达到的预言机复杂度为O(κ̄ᵧ¹⁷ε⁻⁶ρ⁻³)，在额外的有界迭代假设下，其期望预言机复杂度为O(κ̄ᵧ¹⁷ε⁻⁶)。此外，在仅对下层目标施加额外随机平滑假设的情况下，Stoc-SGHA的随机预言机复杂度以高概率提升至O(κ̄ᵧ¹¹ε⁻⁴ρ⁻²)，期望复杂度提升至O(κ̄ᵧ¹¹ε⁻⁴)，与下界的ε依赖关系匹配。
英文摘要
In this work, we study the oracle complexity of finding an $ε$-stationary point for nonconvex-strongly-convex (NC-SC) bilevel optimization using only first-order oracles. Existing methods achieving the best-known complexity guarantees typically rely on double-loop, penalty-based procedures. We propose a novel single-loop algorithm based on a constrained reformulation in which lower-level stationarity is imposed as a constraint. Specifically, we construct a regularized Lagrangian by introducing a quadratic regularizer and restricting the dual variable to a bounded domain, and then apply Smoothed Gradient Descent Ascent [Zhang et al., 2020], with Hessian-vector products approximated via finite differences of gradients. We refer to the resulting deterministic and stochastic algorithms as SGHA and Stoc-SGHA, respectively. In the deterministic setting, SGHA achieves an oracle complexity of $O(\barκ_y^{5}ε^{-2})$, where $\barκ_y$ denotes the relevant condition number. In the stochastic setting, Stoc-SGHA achieves an oracle complexity of $O\left(\barκ_y^{17}ε^{-6}ρ^{-3}\right)$ with probability at least $1-ρ$ for any $ρ\in(0,1)$, and an oracle complexity of $O\left(\barκ_y^{17}ε^{-6}\right)$ in expectation under an additional bounded-iterate assumption. Moreover, under an additional stochastic smoothness assumption imposed only on the lower-level objective, the stochastic oracle complexity of Stoc-SGHA improves to $O\left(\barκ_y^{11}ε^{-4}ρ^{-2}\right)$ with high probability and $O\left(\barκ_y^{11}ε^{-4}\right)$ in expectation, matching the $ε$-dependence of the lower bounds.
