Partial calmness is a celebrated but restrictive property of bilevel optimization problems whose presence opens a way to the derivation of Karush--Kuhn--Tucker-type necessary optimality conditions in order to characterize local minimizers. In the past, sufficient conditions for the validity of partial calmness have been investigated. In this regard, the presence of a linearly structured lower level problem has turned out to be beneficial. However, the associated literature suffers from inaccurate results. In this note, we clarify some regarding erroneous statements and visualize the underlying issues with the aid of illustrative counterexamples.
The bilevel program is an optimization problem where the constraint involves solutions to a parametric optimization problem. It is well-known that the value function reformulation provides an equivalent single-level optimization problem but it results in a nonsmooth optimization problem which never satisfies the usual constraint qualification such as the Mangasarian-Fromovitz constraint qualification (MFCQ). In this paper we show that even the first order sufficient condition for metric subregularity (which is in general weaker than MFCQ) fails at each feasible point of the bilevel program. We…
The partial calmness for the bilevel programming problem (BLPP) is an important condition which ensures that a local optimal solution of BLPP is a local optimal solution of a partially penalized problem where the lower level optimality constraint is moved to the objective function and hence a weaker constraint qualification can be applied. In this paper we propose a sufficient condition in the form of a partial error bound condition which guarantees the partial calmness condition. We analyse the partial calmness for the combined program based on the Bouligand (B-) and the Fritz John (FJ) stati…
In this paper, we exploit the so-called value function reformulation of the bilevel optimization problem to develop duality results for the problem. Our approach builds on Fenchel-Lagrange-type duality to establish suitable results for the bilevel optimization problem. First, we overview some standard duality results to show that they are not applicable to our problem. Secondly, via the concept of partial calmness, we establish weak and strong duality results. In particular, Lagrange, Fenchel-Lagrange, and Toland-Fenchel- Lagrange duality concepts are investigated for this type of problems und…
We consider the optimistic bilevel optimization problem, known to have a wide range of applications in engineering, that we transform into a single-level optimization problem by means of the lower-level optimal value function reformulation. Subsequently, based on the partial calmness concept, we build an equation system, which is parameterized by the corresponding partial exact penalization parameter. We then design and analyze a Levenberg-Marquardt method to solve this parametric system of equations. Considering the fact that the selection of the partial exact penalization parameter is a crit…
In this paper, we propose a combined approach with second-order optimality conditions of the lower level problem to study constraint qualifications and optimality conditions for bilevel programming problems. The new method is inspired by the combined approach developed by Ye and Zhu in 2010, where the authors combined the classical first-order and the value function approaches to derive new necessary optimality conditions. In our approach, we add a second-order optimality condition to the combined program as a new constraint. We show that when all known approaches fail, adding the second-order…
The presence of Lipschitzian properties for solution mappings associated with nonlinear parametric optimization problems is desirable in the context of stability analysis or bilevel optimization. An example of such a Lipschitzian property for set-valued mappings, whose graph is the solution set of a system of nonlinear inequalities and equations, is R-regularity. Based on the so-called relaxed constant positive linear dependence constraint qualification, we provide a criterion ensuring the presence of the R-regularity property. In this regard, our analysis generalizes earlier results of that t…
Bilevel optimization have gained growing interests, with numerous applications found in meta learning, minimax games, reinforcement learning, and nested composition optimization. This paper studies the problem of distributed bilevel optimization over a network where agents can only communicate with neighbors, including examples from multi-task, multi-agent learning and federated learning. In this paper, we propose a gossip-based distributed bilevel learning algorithm that allows networked agents to solve both the inner and outer optimization problems in a single timescale and share information…
Meta；The Ohio State University（俄亥俄州立大学）；University at Buffalo（布法罗大学）
Federated bilevel optimization has received increasing attention in various emerging machine learning and communication applications. Recently, several Hessian-vector-based algorithms have been proposed to solve the federated bilevel optimization problem. However, several important properties in federated learning such as the partial client participation and the linear speedup for convergence (i.e., the convergence rate and complexity are improved linearly with respect to the number of sampled clients) in the presence of non-i.i.d.~datasets, still remain open. In this paper, we fill these gaps…
University of Bath（巴斯大学）；CNRS（法国国家科学研究中心）；Ecole Normale Supérieure, PSL University（巴黎文理研究大学高等师范学院）
Iteratively reweighted least square (IRLS) is a popular approach to solve sparsity-enforcing regression problems in machine learning. State of the art approaches are more efficient but typically rely on specific coordinate pruning schemes. In this work, we show how a surprisingly simple reparametrization of IRLS, coupled with a bilevel resolution (instead of an alternating scheme) is able to achieve top performances on a wide range of sparsity (such as Lasso, group Lasso and trace norm regularizations), regularization strength (including hard constraints), and design matrices (ranging from cor…
Bilevel Learning via Inexact Stochastic Gradient Descent
EPSRC Centre for Doctoral Training in Statistical Applied Mathematics at Bath（巴斯大学统计应用数学博士培训中心）；Australian Research Council（澳大利亚研究理事会）；Indian Institute of Technology (IIT), Kharagpur（印度理工学院（Kharagpur分校））；University of Melbourne（墨尔本大学）；Department of Electronics & Electrical Communication Engineering（电子与电气通信工程系）；School of Mathematics and Statistics（数学与统计学学院）
