arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.04660cs.LG
收藏

一种无Hessian逆的二层优化单循环算法

A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

Electrical and Computer Engineering, University of Pittsburgh(匹兹堡大学电气与计算机工程系);MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Junyi Li, Bin Gu, Heng Huang

摘要:

本文提出了一种新的求解双层优化问题的无Hessian逆的完全单循环算法(FSLA)。经典的两层优化算法承认一个双循环结构,这是计算昂贵的。近年来,人们提出了几种交替优化内外变量的单循环算法。然而,这些算法尚未实现完全的单循环。因为它们忽略了为给定的内部和外部状态评估超梯度所需的循环。为了开发一个完整的单循环算法,我们首先研究了超梯度的结构,并确定了一个通用的近似公式的超梯度计算,其中包括几个以前的常见方法,例如通过时间的反向传播,共轭梯度,\emph{etc.}基于这个公式,我们引入了一个新的状态变量来保持历史的超梯度信息。结合我们的新配方与交替更新的内部和外部变量,我们提出了一个有效的完全单循环算法。我们从理论上证明了新状态产生的误差是有界的,并且我们的算法以$O(ε^{-2})$的速度收敛。最后,我们通过多个基于双层优化的机器学习任务验证了我们的算法的有效性。

英文摘要:

In this paper, we propose a new Hessian inverse free Fully Single Loop Algorithm (FSLA) for bilevel optimization problems. Classic algorithms for bilevel optimization admit a double loop structure which is computationally expensive. Recently, several single loop algorithms have been proposed with optimizing the inner and outer variable alternatively. However, these algorithms not yet achieve fully single loop. As they overlook the loop needed to evaluate the hyper-gradient for a given inner and outer state. In order to develop a fully single loop algorithm, we first study the structure of the hyper-gradient and identify a general approximation formulation of hyper-gradient computation that encompasses several previous common approaches, e.g. back-propagation through time, conjugate gradient, \emph{etc.} Based on this formulation, we introduce a new state variable to maintain the historical hyper-gradient information. Combining our new formulation with the alternative update of the inner and outer variables, we propose an efficient fully single loop algorithm. We theoretically show that the error generated by the new state can be bounded and our algorithm converges with the rate of $O(ε^{-2})$. Finally, we verify the efficacy our algorithm empirically through multiple bilevel optimization based machine learning tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑