Bilevel optimization has arisen as a powerful tool for solving a variety of machine learning problems. Two current popular bilevel optimizers AID-BiO and ITD-BiO naturally involve solving one or two sub-problems, and consequently, whether we solve these problems with loops (that take many iterations) or without loops (that take only a few iterations) can significantly affect the overall computational efficiency. Existing studies in the literature cover only some of those implementation choices, and the complexity bounds available are not refined enough to enable rigorous comparison among different implementations. In this paper, we first establish unified convergence analysis for both AID-BiO and ITD-BiO that are applicable to all implementation choices of loops. We then specialize our results to characterize the computational complexity for all implementations, which enable an explicit comparison among them. Our result indicates that for AID-BiO, the loop for estimating the optimal point of the inner function is beneficial for overall efficiency, although it causes higher complexity for each update step, and the loop for approximating the outer-level Hessian-inverse-vector product reduces the gradient complexity. For ITD-BiO, the two loops always coexist, and our convergence upper and lower bounds show that such loops are necessary to guarantee a vanishing convergence error, whereas the no-loop scheme suffers from an unavoidable non-vanishing convergence error. Our numerical experiments further corroborate our theoretical results.
SPARKLE: A Unified Single-Loop Primal-Dual Framework for Decentralized Bilevel Optimization
Peking University(北京大学);IBM Research(IBM研究院);University of Pennsylvania(宾夕法尼亚大学);National Engineering Labratory for Big Data Analytics and Applications(国家大数据分析与应用工程实验室);AI for Science Institute(科学智能研究所)
Implicit differentiation for fast hyperparameter selection in non-smooth convex learning
Université Paris-Saclay(巴黎萨克雷大学);Inria(法国国家信息与自动化研究所);CEA(法国原子能和替代能源委员会);Université de Bourgogne(勃艮第大学);Institut Mathématique de Bourgogne(勃艮第数学研究所);Università degli Studi di Genova(热那亚大学);Google Research(谷歌研究院);CNRS(法国国家科学研究中心);Université de Montpellier(蒙彼利埃大学)
When Does Preconditioning Help or Hurt Generalization?
University of Tokyo(东京大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);Columbia University(哥伦比亚大学);RIKEN Center for Brain Science(理化学研究所脑科学中心);University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(向量人工智能研究所);Google Research(谷歌研究院)
Permutation Randomization on Nonsmooth Nonconvex Optimization: A Theoretical and Experimental Study
School of Computer and Cyber Sciences(计算机与电子科学学院);Augusta University(奥古斯塔大学);Department of Graduate Psychology(研究生心理学系);James Madison University(詹姆斯·麦迪逊大学);School of Computing(计算学院);University of Georgia(佐治亚大学)
Preconditioning Benefits of Spectral Orthogonalization in Muon
Penn Department of Statistics and Data Science, Wharton School, University of Pennsylvania(宾夕法尼亚大学统计与数据科学系,沃顿商学院,宾夕法尼亚大学);Yale Department of Statistics and Data Science, Yale University(耶鲁大学统计与数据科学系,耶鲁大学);Department of Electrical and Systems Engineering, University of Pennsylvania(宾夕法尼亚大学电气与系统工程系,宾夕法尼亚大学)