双层优化与下层压缩映射：无需热启动的最优样本复杂度
Bilevel Optimization with a Lower-level Contraction: Optimal Sample Complexity without Warm-start
AI总结：
本文针对无热启动的双层优化问题，提出下层随机不动点迭代与上层投影非精确梯度下降的简单方法，在随机和确定性设置下分别实现$O(ε^{-2})$和$\ ilde{O}(ε^{-1})$的阶数最优样本复杂度，并简化了分析。
摘要：
我们分析了一类一般的双层问题，其中上层问题是最小化一个光滑目标函数，下层问题是寻找一个光滑压缩映射的不动点。这类问题包括元学习、均衡模型、超参数优化和数据投毒对抗攻击等实例。近期的几项工作提出了对下层问题进行热启动的算法，即使用前一次下层的近似解作为下层求解器的起始点。这种热启动过程可以在随机和确定性设置中提高样本复杂度，在某些情况下达到阶数最优的样本复杂度。然而，在元学习和均衡模型等情形下，热启动过程并不适用或效果不佳。在本工作中，我们证明在没有热启动的情况下，仍然可以实现阶数（接近）最优的样本复杂度。具体而言，我们提出了一种简单方法，在下层使用（随机）不动点迭代，在上层使用投影非精确梯度下降，在随机和确定性设置中分别使用$O(ε^{-2})$和$\ ilde{O}(ε^{-1})$个样本即可达到$ε$-稳定点。最后，与使用热启动的方法相比，我们的方法提供了更简单的分析，无需研究上层和下层迭代之间的耦合交互。
英文摘要：
We analyse a general class of bilevel problems, in which the upper-level problem consists in the minimization of a smooth objective function and the lower-level problem is to find the fixed point of a smooth contraction map. This type of problems include instances of meta-learning, equilibrium models, hyperparameter optimization and data poisoning adversarial attacks. Several recent works have proposed algorithms which warm-start the lower-level problem, i.e.~they use the previous lower-level approximate solution as a staring point for the lower-level solver. This warm-start procedure allows one to improve the sample complexity in both the stochastic and deterministic settings, achieving in some cases the order-wise optimal sample complexity. However, there are situations, e.g., meta learning and equilibrium models, in which the warm-start procedure is not well-suited or ineffective. In this work we show that without warm-start, it is still possible to achieve order-wise (near) optimal sample complexity. In particular, we propose a simple method which uses (stochastic) fixed point iterations at the lower-level and projected inexact gradient descent at the upper-level, that reaches an $ε$-stationary point using $O(ε^{-2})$ and $\tilde{O}(ε^{-1})$ samples for the stochastic and the deterministic setting, respectively. Finally, compared to methods using warm-start, our approach yields a simpler analysis that does not need to study the coupled interactions between the upper-level and lower-level iterates.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
