LancBiO：通过 Krylov 子空间的动态 Lanczos 辅助双层优化
LancBiO: dynamic Lanczos-aided bilevel optimization via Krylov subspace
AI总结：
针对双层优化中 Hessian 逆计算瓶颈，提出基于 Lanczos 过程的动态 Krylov 子空间方法，以较低代价近似超梯度，实现 $\mathcal{O}(ε^{-1})$ 收敛并在合成与深度学习任务中验证高效性。
摘要：
双层优化在机器学习中具有广泛的应用，其结构具有复杂的层次性。基于梯度的方法已成为解决大规模双层问题的常见途径。然而，超梯度的计算涉及 Hessian 逆向量乘积，这限制了效率并被视为瓶颈。为了规避求逆，我们借助 Lanczos 过程构造了一系列低维近似 Krylov 子空间。由此，所构造的子空间能够以较少的计算量动态且增量地近似 Hessian 逆向量乘积，从而获得对超梯度的有利估计。此外，我们提出了一种可证明的基于子空间的双层问题框架，其中一个核心步骤是求解一个小规模的三对角线性系统。据我们所知，这是子空间技术首次被引入双层优化。这一成功的尝试不仅享有 $\mathcal{O}(ε^{-1})$ 的收敛速率，而且在一个合成问题和两个深度学习任务中展示了其高效性。
英文摘要：
Bilevel optimization, with broad applications in machine learning, has an intricate hierarchical structure. Gradient-based methods have emerged as a common approach to large-scale bilevel problems. However, the computation of the hyper-gradient, which involves a Hessian inverse vector product, confines the efficiency and is regarded as a bottleneck. To circumvent the inverse, we construct a sequence of low-dimensional approximate Krylov subspaces with the aid of the Lanczos process. As a result, the constructed subspace is able to dynamically and incrementally approximate the Hessian inverse vector product with less effort and thus leads to a favorable estimate of the hyper-gradient. Moreover, we propose a provable subspace-based framework for bilevel problems where one central step is to solve a small-size tridiagonal linear system. To the best of our knowledge, this is the first time that subspace techniques are incorporated into bilevel optimization. This successful trial not only enjoys $\mathcal{O}(ε^{-1})$ convergence rate but also demonstrates efficiency in a synthetic problem and two deep learning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
