arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.14224cs.LGmath.OCstat.ML
收藏

Bilevel优化器会从循环中受益吗

Will Bilevel Optimizers Benefit from Loops

University of Michigan(密歇根大学);George Mason University(乔治梅森大学);The Ohio State University(俄亥俄州立大学)

Kaiyi Ji, Mingrui Liu, Yingbin Liang, Lei Ying

摘要:

双层优化已经成为解决各种机器学习问题的有力工具。两个当前流行的双层优化器AID-BiO和ITD-BiO自然涉及解决一个或两个子问题,因此,我们是否使用循环(需要多次迭代)或不使用循环(只需要几次迭代)来解决这些问题可以显着影响整体计算效率。文献中现有的研究只涵盖了其中的一些实现选择,并且可用的复杂性界限还不够精细,无法在不同的实现之间进行严格的比较。在本文中,我们首先建立了统一的收敛性分析AID-BiO和ITD-BiO适用于所有的实现选择的循环。然后,我们专门我们的结果来表征所有实现的计算复杂性,这使得它们之间的显式比较。我们的结果表明,对于AID-BiO,用于估计内部函数的最优点的循环有利于整体效率,尽管它会导致每个更新步骤的复杂度更高,并且用于近似外层Hessian逆向量积的循环降低了梯度复杂度。对于ITD-BiO,这两个循环总是共存的,我们的收敛上界和下界表明,这样的循环是必要的,以保证消失的收敛误差,而无循环计划遭受不可避免的非零收敛误差。我们的数值实验进一步证实了我们的理论结果。

英文摘要:

Bilevel optimization has arisen as a powerful tool for solving a variety of machine learning problems. Two current popular bilevel optimizers AID-BiO and ITD-BiO naturally involve solving one or two sub-problems, and consequently, whether we solve these problems with loops (that take many iterations) or without loops (that take only a few iterations) can significantly affect the overall computational efficiency. Existing studies in the literature cover only some of those implementation choices, and the complexity bounds available are not refined enough to enable rigorous comparison among different implementations. In this paper, we first establish unified convergence analysis for both AID-BiO and ITD-BiO that are applicable to all implementation choices of loops. We then specialize our results to characterize the computational complexity for all implementations, which enable an explicit comparison among them. Our result indicates that for AID-BiO, the loop for estimating the optimal point of the inner function is beneficial for overall efficiency, although it causes higher complexity for each update step, and the loop for approximating the outer-level Hessian-inverse-vector product reduces the gradient complexity. For ITD-BiO, the two loops always coexist, and our convergence upper and lower bounds show that such loops are necessary to guarantee a vanishing convergence error, whereas the no-loop scheme suffers from an unavoidable non-vanishing convergence error. Our numerical experiments further corroborate our theoretical results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑