Bilevel optimization is widely applied in many machine learning tasks such as hyper-parameter learning, meta learning and reinforcement learning. Although many algorithms recently have been developed to solve the bilevel optimization problems, they generally rely on the (strongly) convex lower-level problems. More recently, some methods have been proposed to solve the nonconvex-PL bilevel optimization problems, where their upper-level problems are possibly nonconvex, and their lower-level problems are also possibly nonconvex while satisfying Polyak-Łojasiewicz (PL) condition. However, these methods still have a high convergence complexity or a high computation complexity such as requiring compute expensive Hessian/Jacobian matrices and its inverses. In the paper, thus, we propose an efficient Hessian/Jacobian-free method (i.e., HJFBiO) with the optimal convergence complexity to solve the nonconvex-PL bilevel problems. Theoretically, under some mild conditions, we prove that our HJFBiO method obtains an optimal convergence rate of $O(\frac{1}{T})$, where $T$ denotes the number of iterations, and has an optimal gradient complexity of $O(ε^{-1})$ in finding an $ε$-stationary solution. We conduct some numerical experiments on the bilevel PL game and hyper-representation learning task to demonstrate efficiency of our proposed method.
On the Complexity of Finding Stationary Points in Nonconvex Simple Bilevel Optimization
Department of Electrical and Computer Engineering, The University of Texas at Austin(电气与计算机工程系,德克萨斯大学奥斯汀分校);Department of Systems and Industrial Engineering, The University of Arizona(系统与工业工程系,亚利桑那大学)
SUN-DSBO: A Structured Unified Framework for Nonconvex Decentralized Stochastic Bilevel Optimization
Southern University of Science and Technology(南方科技大学);Pengcheng Laboratory(鹏城实验室);Sun Yat-sen University(中山大学);National Center for Applied Mathematics Shenzhen(深圳应用数学国家中心)
Hessian-Free Distributed Bilevel Optimization via Penalization with Time-Scale Separation
State Key Laboratory of Industrial Control Technology and the College of Control Science and Engineering, Zhejiang University(浙江大学工业控制技术国家重点实验室与控制科学与工程学院);School of Electrical Engineering and Computer Science, Pennsylvania State University(宾夕法尼亚州立大学电气与计算机工程学院)
Adaptive Algorithms for Nonconvex Bilevel Optimization under PŁ Conditions
School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University(复旦大学当代应用数学上海市重点实验室)
A Hybrid Subgradient Method for Nonsmooth Nonconvex Bilevel Optimization
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen University(深圳大学);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学)
Public Health School, Fujian Medical University(福建医科大学公共卫生学院);School of Mathematics and Statistics, Fuzhou University(福州大学数学与统计学院);Center for Applied Mathematics of Fujian Province(福建省应用数学中心);National Center for Applied Mathematics Shenzhen, SUSTech(南方科技大学深圳国家应用数学中心);Department of Mathematics, SUSTech(南方科技大学数学系);School of Science, BUPT(北京邮电大学理学院);The Smart City Research Institute of CETC(中国电子科技集团公司智慧城市研究院)
Bilevel Learning with Inexact Stochastic Gradients
Department of Mathematical Sciences, University of Bath(巴斯大学数学科学系);Department of Electronics & Electrical Communication Engineering, Indian Institute of Technology (IIT) Kharagpur(印度理工学院Kharagpur电子与电气通信工程系);School of Mathematics and Statistics, University of Sydney(悉尼大学数学与统计学学院)
Moreau Envelope for Nonconvex Bi-Level Optimization: A Single-loop and Hessian-free Solution Strategy
School of Software Technology, Dalian University of Technology(大连理工大学软件学院);Pazhou Lab(琶洲实验室);National Center for Applied Mathematics Shenzhen, Southern University of Science and Technology(南方科技大学深圳应用数学中心);Department of Mathematics, Southern University of Science and Technology(南方科技大学数学系);Department of Mathematics and Statistics, University of Victoria(维多利亚大学数学与统计系)
A Regularized Newton Method for Nonconvex Optimization with Global and Local Complexity Guarantees
Tsinghua University(清华大学);The Hong Kong Polytechnic University(香港理工大学);Beijing Institute of Mathematical Sciences and Applications(北京数学科学及应用研究所);Chinese Academy of Sciences(中国科学院)
Universal heavy-ball method for nonconvex optimization under Hölder continuous Hessians
Graduate School of Information Science and Technology, University of Tokyo(东京大学信息理工学专攻);Center for Advanced Intelligence Project, RIKEN(理化学研究所先进智能项目中心)